Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big...
Transcript of Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big...
![Page 1: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/1.jpg)
Esri UC2013 . Technical Workshop . EsrEsrEsrEsrEsrEsrEsrsrsrrri Uii Ui Ui Ui Ui Uii UUi C20C20C20C20CCC 00C20CC2 131313 13 13 13 3 . T. T. TTT. T. T. TTTTTTechechechechhecechchchec nicnicnicnicnicnicninicnicnicicalalalalalalalalalallla WorWorWorWorWorWorWorWorWorWoorWo kshkshkshkkshkshkshkkshkskkshkshop op op opop oop op opp . .. . . .
Technical Workshop
2013 Esri International User Conference July 8–12, 2013 | San Diego, California
Big Data Spatial Analytics – An Introduction
Marwa Mabrouk Mansour Raad
![Page 2: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/2.jpg)
Esri UC2013 . Technical Workshop .
Agenda
• What is Big Data? • What is Hadoop? • How does Spatial integrate with Big Data and Hadoop?
• How do I get started?
Big Data Spatial Analytics – An Introduction
![Page 3: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/3.jpg)
Esri UC2013 . Technical Workshop .
New Challenges for Organizations
• Better Decision making • Intelligence • Insight/ Foresight
• Social Data Analysis • Log files analysis • Fraud Detection
Big Data Spatial Analytics – An Introduction
![Page 4: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/4.jpg)
Esri UC2013 . Technical Workshop .
Collect Data!!!
Big Data Spatial Analytics – An Introduction
![Page 5: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/5.jpg)
Esri UC2013 . Technical Workshop .
What is Big Data???
Big Data Spatial Analytics – An Introduction
Is it Satellite & Aerial Imagery???
Is it a large network of a million points???
Is it a simple Geodatabase with Billions of records???
Is it this technology called hadoop?
![Page 6: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/6.jpg)
Esri UC2013 . Technical Workshop .
Big Data, a new Data Type
• Volume - Ever growing data, petabytes / zetabytes - Use 350 billion annual meter readings to better predict power
consumption
• Velocity - Time-sensitive, two mins can be too late - Turn 12 TBs of tweets into updates about people’s conditions
during a storm for emergency response
• Variety - Any type of data, text / audio/ video / click streams / other - Exploit documents, images, voice recordings, videos in customer
experience analysis programs to improve customer satisfaction
Big Data
Big Data Spatial Analytics – An Introduction
![Page 7: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/7.jpg)
Esri UC2013 . Technical Workshop .
What is hadoop?
Big Data Spatial Analytics – An Introduction
HDFS
MapReduce
Common
![Page 8: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/8.jpg)
Esri UC2013 . Technical Workshop .
![Page 9: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/9.jpg)
Esri UC2013 . Technical Workshop .
![Page 10: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/10.jpg)
Esri UC2013 . Technical Workshop .
• Parallel Framework • Executes Map Reduce Task • Reads HDFS • Java/Python/Awk
![Page 11: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/11.jpg)
Esri UC2013 . Technical Workshop .
$> cat input | Map | sort | Reduce > out
![Page 12: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/12.jpg)
Esri UC2013 . Technical Workshop .
![Page 13: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/13.jpg)
Esri UC2013 . Technical Workshop .
Apache Hive
![Page 14: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/14.jpg)
Esri UC2013 . Technical Workshop .
SQL
MapReduce Job
![Page 15: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/15.jpg)
Esri UC2013 . Technical Workshop .
HDFS CSV TSV JSON BINARY
MapReduce
hive> select * from cities where country= lebanon ;
![Page 16: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/16.jpg)
Esri UC2013 . Technical Workshop .
What About Spatial ?
![Page 17: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/17.jpg)
Esri UC2013 . Technical Workshop .
Big Data – A new data type for Geospatial
Imagery
DBMS
Services Sensor
Networks
Big Data
Spreadsheets Maps
Imagery
Sp
Social Media
SS
Big Data Spatial Analytics – An Introduction
![Page 18: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/18.jpg)
Esri UC2013 . Technical Workshop .
Geospatial in Big Data
• Big Data with location in Enterprise Big Data warehouse
• Run spatial queries on data where it resides
• Pull Results in ArcGIS: Visualize results as a map; Include in a report; Publish in a web or mobile app
Big Data
Big Data Spatial Analytics – An Introduction
![Page 19: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/19.jpg)
Esri UC2013 . Technical Workshop .
GIS Tools for Hadoop
• http://esri.github.com/gis-tools-for-hadoop/ • Support Running geometry-based spatial queries inside Hadoop
• Open Source - Apache 2.0 license
• Two types of users - Developers - ArcGIS users
Big Data Spatial Analytics – An Introduction
![Page 20: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/20.jpg)
Esri UC2013 . Technical Workshop .
Run Hive Queries with spatial
operators
013 1 . Technical Workshop .
Build Map/ Reduce Spatial
Apps in Java
ArcGIS
Esri Geometry API
Spatial Framework
Geo
GIS Tools for Hadoop (Use samples with Hadoop & Hive to get started)
ArcGIS Connect From ArcGIS
to Hadoop using GP
Geoprocessing
Tools
GIS tools for Hadoop libraries
![Page 21: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/21.jpg)
Esri UC2013 . Technical Workshop .
GIS Tools for Hadoop Github repos
Esri Geometry API for Java (EGA)
Spatial framework for Hadoop
GeoProcessing tools for Hadoop
GIS tools for Hadoop
r J a
Developer
Starting Point
ArcGIS User
Starting Point
http://esri.github.com/gis-tools-for-hadoop/
![Page 22: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/22.jpg)
Esri UC2013 . Technical Workshop .
Spatial Storage
• CSV,TSV Lat,Lon • Esri JSON format
- {geometry:{x:-123,y:45},attributes:{}}
• Custom
![Page 23: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/23.jpg)
Esri UC2013 . Technical Workshop .
User Defined Functions
• select tolower( ESRI ); • select * from mytable where cos(rad) < 0.1;
![Page 24: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/24.jpg)
Esri UC2013 . Technical Workshop .
Spatial UDF !
![Page 25: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/25.jpg)
Esri UC2013 . Technical Workshop .
select * from cities where near(x,y,-84.2,39.4);
![Page 26: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/26.jpg)
Esri UC2013 . Technical Workshop .
select * from cities where contains(x,y, #mypolys );
![Page 27: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/27.jpg)
Esri UC2013 . Technical Workshop .
How to get started? Technology
• Download GIS Tools for Hadoop -Sample (jars + sample data/ query)
• Hadoop cluster -Amazon Web Services EMR -Cloudera Manager and VM, CDH
• Run sample
Big Data Spatial Analytics – An Introduction
![Page 28: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/28.jpg)
Esri UC2013 . Technical Workshop .
How to get started? Data and Query
• Use Case, problem definition
• Dataset - Formats - Conversion - Location - Loading
• Query - Hive Script
Big Data Spatial Analytics – An Introduction
Big Data
![Page 29: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/29.jpg)
Esri UC2013 . Technical Workshop . Esri UC2013 . Technical Workshop . Esri UC2013 . Technical Workshop .
AWS Elastic Map Reduce GP Tools
Cloud
Big Data Spatial Analytics – An Introduction
![Page 30: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/30.jpg)
Esri UC2013 . Technical Workshop . Esri UC2013 . Technical Workshop . Esri UC2013 . Technical Workshop .
Cloudera Map Reduce
On Premise
Big Data Spatial Analytics – An Introduction
![Page 31: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/31.jpg)
Esri UC2013 . Technical Workshop .
Big Data The Data warehouse
Big Data Spatial Analytics – An Introduction
![Page 32: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/32.jpg)
Esri UC2013 . Technical Workshop . Big Data Spatial Analytics – An Introduction Esri UC2013 . Technical Workshop . Big Data Spatial Analytics – An Introduction
![Page 33: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/33.jpg)
Esri UC2013 . Technical Workshop .
Big Data Partners
Big Data Spatial Analytics – An Introduction
![Page 34: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/34.jpg)
Esri UC2013 . Technical Workshop .
Marwa Mabrouk - [email protected] Mansour Raad - [email protected], http://thunderheadxpler.blogspot.com/ Please fill out the session evaluation
First Offering ID: 2333
Online – www.esri.com/ucsessionsurveys Paper – pick up and put in drop box
Thank you
![Page 35: Big Data Spatial Analytics – An Introduction...Big Data Spatial Analytics – An Introduction Big Data Esri UC2013 . Technical Workshop . Esri UEsri UC2013 . Technical Workshop .C2013](https://reader033.fdocuments.net/reader033/viewer/2022053001/5f05b0d97e708231d41437cf/html5/thumbnails/35.jpg)
Esri UC2013 . Technical Workshop . Big Data Spatial Analytics – An Introduction