Lost In The Clouds
-
Upload
georgejames -
Category
Technology
-
view
984 -
download
4
description
Transcript of Lost In The Clouds
![Page 1: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/1.jpg)
Lost in the Clouds
An example of a schemaless database application in the cloud
SlipstreamUSA :: March 25, 2009
![Page 2: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/2.jpg)
The Project
OpenStreetMap A free map of the planet
Free geospatial data Licensed under CC-BY-SA
Created from Crowd-sourced contributions 90,000 registered users 1,000+ contributors per day
Used for: Wikipedia Flikr OpenCycleMap.org OpenPisteMap.org YourNavigation.org Mobile devices
© Constantin Litvak
![Page 3: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/3.jpg)
Coverage
CC-BY-SA 2.0 © OpenStreetMap
![Page 4: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/4.jpg)
The OpenStreetMap Database Server
In the cloud MySQL Ways: 24,956,171
Nodes: 305,956,583
Tags: 3.6 billion highway=motorway name=Broadway foo=bar
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
![Page 5: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/5.jpg)
Interfaces
API For updates
(1,000 users per day) Queries of up to
0.25² degrees Weekly Planet Dump
100Gbyte XML file Changes
Feed of changes Daily / Hourly / Minutely One file per minute
(~30Kb compressed)CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
![Page 6: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/6.jpg)
Problem Statement
ProblemWant to be able to query the data in bigger chunks and to do it more selectively
SolutionCloud based schemaless database with simple to use query interface and lots of indexes
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
![Page 7: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/7.jpg)
XAPI Service
GT.M or Cache 250Gbyte database
Nodes: 30Gb Node tags: 115Gb Node indexes: 70Gb Ways: 21Gb Way tags: 10Gb Way indexes: 18Gb
Apache + serverLink Licensed as AGPL
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
![Page 8: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/8.jpg)
XAPI Service
In the cloud Queries of up to 100² degrees Query by tags and tag values Scaleable REST style interface
http://xapi.openstreetmap.org/api/0.6/*[tourism=hotel][bbox=-75.7,40.0,-75.4,40.2]
Import planet dump Import minute diff
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
![Page 9: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/9.jpg)
XAPI Service
adaabcdcabaadab
Geospatial Index – quadstrings
![Page 10: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/10.jpg)
XAPI Service
^way(27016525)="adaabcdcabaadab"^way(27016525,1)=296138118^way(27016525,2)=296138119^way(27016525,3)=296138120^way(27016525,4)=296138121^way(27016525,5)=296138118
^waytag(27016525,"addr:housenumber")=2^waytag(27016525,"building")="yes"
^wayx("building","*","adaabcdcabaadab",27016525)=""^wayx("building","*","adaabcdcabaadab",27028298)=""^wayx("building","*","adaabcdcabaadab",27028299)=""^wayx("building","*","adaabcdcabaadab",27028326)=""^wayx("building","*","adaabcdcabaadab",27028327)=""^wayx("building","*","adaabcdcabaadab",27035972)=""^wayx("building","*","adaabcdcabaadab",27035973)=""^wayx("building","*","adaabcdcabaadab",27035974)=""^wayx("building","*","adaabcdcabaadab",27035975)=""^wayx("building","*","adaabcdcabaadab",27035984)="“
<way id='27016525'><way id='27016525'> <nd ref='296138118'/><nd ref='296138118'/> <nd ref='296138119'/><nd ref='296138119'/> <nd ref='296138120'/><nd ref='296138120'/> <nd ref='296138121'/><nd ref='296138121'/> <nd ref='296138118'/><nd ref='296138118'/> <tag k='addr:housenumber' v='2'/><tag k='addr:housenumber' v='2'/> <tag k='building' v='yes'/><tag k='building' v='yes'/> </way></way>
![Page 11: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/11.jpg)
Deployment
Currently 5 instances: London Paris San Diego Zurich Amsterdam
(Wikimedia-de) More servers needed:
Internet connectivity 300Gb disk storage
CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
![Page 12: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/12.jpg)
Wikipedia: Query to Map
{{osm-query |way=name=Arizona Canal}}
CC-BY-SA 2.0 © OpenStreetMap
![Page 13: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/13.jpg)
Internals
Debian Apache web-server REST
URI represents a resource
Xpath flavoring serverLink
WebLink emulator GT.M or Cache
POM AGPL
![Page 14: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/14.jpg)
XAPI Future
Load Sharing / balancing More Xpath-like queries JSON output Generic indexing service More servers
![Page 15: Lost In The Clouds](https://reader034.fdocuments.net/reader034/viewer/2022042518/549e6074ac79594c768b472c/html5/thumbnails/15.jpg)
CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap
Thank you
Thank youQuestions?