SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.
-
Upload
joan-short -
Category
Documents
-
view
280 -
download
6
Transcript of SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.
![Page 1: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/1.jpg)
SpatialHadoop:A MapReduce Framework
for Spatial Data
汇报人:赵郁亮 2015-8-3 ICDE 2015
![Page 2: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/2.jpg)
Executive Summary
• Propose a full-fledged MapReduce framework with native support for spatial data.
• Propose a new system architecture with fourlayers:language,operations,mapreduce and storage layers.
• SpatialHadoop achieve orders of magnitude better performance than hadoop for spatial data processing.
![Page 3: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/3.jpg)
Outline
• Introduction
• Related work
• SpatialHadoop Architecture
• Experiments
![Page 4: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/4.jpg)
Introduction
• An explosion in the amounts of spatial data were produced by various devices such as smart phones,satellites,and medical devices.
• Hadoop was adopted as a solution for scalable processing of huge datasets in many applications,e.g.,machine learning ,graph processing and behavioral simulations.
• ESRI has released ‘GIS Tools on Hadoop’.
![Page 5: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/5.jpg)
Introduction
• Parallel-Secondo
• MD-HBase
• Hadoop-GIS
• SpatialHadoop
![Page 6: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/6.jpg)
Related work
• Specific spatial operations
R-tree construction
Range query
kNN query
All NN query
• Systems
Hadoop-GIS
MD-Hbase
Parallel-Secondo
![Page 7: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/7.jpg)
SpatialHadoop Architecture
![Page 8: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/8.jpg)
SpatialHadoop Architecture
• Language Layer(Pigeon)
Data types
Spatial functions
KNN query
![Page 9: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/9.jpg)
SpatialHadoop Architecture
• Storage Layer(Indexing) Existing techniques for spatial indexing in
Hadoop
1) Build only
2 ) Custom on-the-fly indexing
3) Indexing in HDFS
![Page 10: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/10.jpg)
SpatialHadoop Architecture
• Storage Layer(Indexing) Overview of Indexing in SpatialHadoop
![Page 11: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/11.jpg)
SpatialHadoop Architecture
Index Building
1)Partitioning
Step1:Number of partitions.
Step2:Partitions boundaries.
Step3:Physical partitioning
2)Local Indexing
3)Global Indexing
![Page 12: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/12.jpg)
SpatialHadoop Architecture
Grid file
![Page 13: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/13.jpg)
SpatialHadoop Architecture
R-tree
![Page 14: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/14.jpg)
SpatialHadoop Architecture
R+-tree
![Page 15: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/15.jpg)
SpatialHadoop Architecture
• MapReduce Layer
![Page 16: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/16.jpg)
SpatialHadoop Architecture
• Operations Layer Range QueryKNN
![Page 17: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/17.jpg)
SpatialHadoop Architecture
• Operations Layer Spatial Join
Step1:Global join
Step2:Local join
Step3:Duplicate avoidance
![Page 18: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/18.jpg)
Experiments
• DataSet
TIGER:spatial features in the US such as streets and rivers(60G).
OSM:OpenStreetMap(60G)
NASA:120 Billion(4.6 TB)
SYNTH:2 Billion(128 GB,uniform distribution)
• Experiment Environment
Amazon EC2 cluster of up to 100 nodes
Hadoop 1.2.1 on java 1.6
![Page 19: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/19.jpg)
Experiments
• Evaluation Range Query
![Page 20: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/20.jpg)
Experiments
• Evaluation Range Query
![Page 21: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/21.jpg)
Experiments
• Evaluation KNN
![Page 22: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/22.jpg)
Experiments
• Evaluation Spatial Join
![Page 23: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/23.jpg)
Experiments
• Evaluation Index Creation
![Page 24: SpatialHadoop:A MapReduce Framework for Spatial Data 汇报人:赵郁亮 2015-8-3 ICDE 2015.](https://reader033.fdocuments.net/reader033/viewer/2022061419/56649ecd5503460f94bda39b/html5/thumbnails/24.jpg)