Scaling GIS Data in Non-relational Data Stores
-
Upload
mike-malone -
Category
Technology
-
view
28.967 -
download
0
description
Transcript of Scaling GIS Data in Non-relational Data Stores
![Page 1: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/1.jpg)
Scaling GIS Data in Nonrelational Data Stores
featuring Mike Malone
Tuesday, March 30, 2010
![Page 2: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/2.jpg)
Mike Malone@mjmalone
Tuesday, March 30, 2010
![Page 3: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/3.jpg)
Tuesday, March 30, 2010
![Page 4: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/4.jpg)
SimpleGeoScalable turnkey location infrastructure
Allows you to easily add geo-aware features to an existing application
That result: we need to store and query lots of data (data set is already approaching 1TB, and we haven’t launched)
Tuesday, March 30, 2010
![Page 5: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/5.jpg)
Scaling HTTP is easyNo shared state - shared-nothing architecture• HTTP requests contain all of the information
necessary to generate a response
• HTTP responses contain all of the information necessary for clients to interpret them
• In other words, requests are self-contained and different requests can be routed to different servers
Uniform interface - allows middleware applications to proxy requests, creating a tiered architecture and making load balancing trivial
Tuesday, March 30, 2010
![Page 6: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/6.jpg)
So what’s the problem? Individual HTTP requests have no shared state, but the applications that communicate via HTTP can and do
Application state has to live somewhere• Path of least resistance is usually a relational
database
• But RDBMSs aren’t always the best tool for the job
Tuesday, March 30, 2010
![Page 7: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/7.jpg)
Desirable Data Store Characteristics
Massively distributed
Horizontally scalable
Fault tolerant
Fast
Always available
Tuesday, March 30, 2010
![Page 8: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/8.jpg)
Relational DatabasesBased on the “relational model” first proposed by E.F. Codd in 1969
Tons of implementation experience and lots of robust open source and proprietary implementations
Tuesday, March 30, 2010
![Page 9: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/9.jpg)
RDBMS StrenghtsTheoretically pure
Clean abstraction
Declarative syntax
Mostly standardized
Easy to reason about data
Tuesday, March 30, 2010
![Page 10: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/10.jpg)
ACIDAtomicity - if one part of a transaction fails, the entire transaction fails
Consistency - all data constraints must be met for a transaction to be successful
Isolation - other operations can’t see a transaction that has not yet completed
Durability - once the client has been notified that a transaction succeeded, the transaction will not be lost
Tuesday, March 30, 2010
![Page 11: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/11.jpg)
RDBMS WeaknessesSQL is opaque, and query parsers don’t always do the right thing• Geospatial SQL is particularly bad
The best ones are crazy expensive
Really bad at scaling writes
Strong consistency requirements make horizontal scaling difficult
Tuesday, March 30, 2010
![Page 12: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/12.jpg)
RDBMS WritesRelational databases almost always use B-Tree (or some other tree-based) indexes
Writes are typically implemented by doing an in-place update on disk• Requires random seek to a specific location on
disk
• May require additional seeks to read indexes if they outgrow the disk cache
Disk seeks are bad.
Tuesday, March 30, 2010
![Page 13: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/13.jpg)
CAP Theorem
There are three desirable characteristics of a shared data system that is deployed in a
distributed environment like the web.
Tuesday, March 30, 2010
![Page 14: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/14.jpg)
CAP Theorem1. Consistency - every node in the system contains the same data (e.g., replicas are never out of date)
2. Availability - every request to a non-failing node in the system returns a response
3. Partition Tolerance - system properties (consistency and/or availability) hold even when the system is partitioned and data is lost
Tuesday, March 30, 2010
![Page 15: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/15.jpg)
CAP Theorem
Choose two.
Tuesday, March 30, 2010
![Page 16: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/16.jpg)
Node A Node B
Client
reads & writes
replicates
reads & writes
Tuesday, March 30, 2010
![Page 17: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/17.jpg)
Node A Node B
Client
writes
replicates
Tuesday, March 30, 2010
![Page 18: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/18.jpg)
Node A Node B
Client
responds
acknowledges
Tuesday, March 30, 2010
![Page 19: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/19.jpg)
Node A Node B
Client
responds
o noes!
Tuesday, March 30, 2010
![Page 20: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/20.jpg)
What now?
1. Write fails: data store is unavailable
2. Write succeeds on Node A: data is inconsistent
Tuesday, March 30, 2010
![Page 21: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/21.jpg)
RDBMS ConsistencyRelational databases prioritize consistency
Large scale distributed systems need to be highly available• As we add servers, the possibility of a network
partition or node failure becomes an inevitability
We could write an abstraction layer on top of a relational data store that trades consistency for availability
Or we could switch to a data store that prioritizes the characteristics we really want
Tuesday, March 30, 2010
![Page 22: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/22.jpg)
Nonrelational DBs
• CouchDB
• Cassandra
• Dynamo
• BigTable
• Riak
• Redis
• MongoDB
• SimpleDB
• Memcached
• MemcacheDB
Over the past couple years, a number of specialized data stores have emerged
Tuesday, March 30, 2010
![Page 23: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/23.jpg)
Also Known As NoSQLNot entirely appropriate, since SQL can be implemented on non-relational DBs
But SQL is an opaque abstraction with lots of features that are difficult or impossible to efficiently distribute
Tuesday, March 30, 2010
![Page 24: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/24.jpg)
So what’s different?Most “non-relational” stores specifically emphasize partition tolerance and availability
Typically provide a more relaxed guarantee of eventually consistent
Tuesday, March 30, 2010
![Page 25: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/25.jpg)
NoACID
Tuesday, March 30, 2010
![Page 26: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/26.jpg)
BASE
Basically Available
Soft State
Eventually Consistent
Tuesday, March 30, 2010
![Page 27: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/27.jpg)
Eventual ConsistencyWrite operations are attempted on n nodes that are “authoritative” for the provided key
In the event of a network partition, data is written to another node in the cluster
When the network heals and nodes become available again, inconsistent data is updated
Tuesday, March 30, 2010
![Page 28: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/28.jpg)
SimpleGeo ♥ CassandraNo single point of failure
Efficient online cluster rebalancing allows for incremental scalability
Emphasizes availability and partition tolerance• Eventually consistent
• Tradeoff between consistency and latency exposed to the client
Battle tested - large clusters at Facebook, Digg, and Twitter
Tuesday, March 30, 2010
![Page 29: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/29.jpg)
Cassandra Data ModelColumn - a tuple containing a name, value, and timestamp
Column Family - a group of columns that are stored together on disk
Row - identifier for a specific group of columns in a column family
Super Column - a column that has columns
Tuesday, March 30, 2010
![Page 30: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/30.jpg)
Cassandra Data Model{ '9xj5ss824mzyv.12345': { 'Record': { 'lat': 40.0149856, 'lon': -105.2705456, 'city': 'Boulder', 'state': 'CO' }, }, 'dr5regy3zcfgr.67890': { 'Record': { 'lat': 40.7142691, 'lon': -74.0059729, 'city': 'New York', 'state': 'NY' } }}
Tuesday, March 30, 2010
![Page 31: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/31.jpg)
Cassandra Data Model{ '9xj5ss824mzyv.12345': { 'Record': { 'lat': 40.0149856, 'lon': -105.2705456, 'city': 'Boulder', 'state': 'CO' }, }, 'dr5regy3zcfgr.67890': { 'Record': { 'lat': 40.7142691, 'lon': -74.0059729, 'city': 'New York', 'state': 'NY' } }}
Tuesday, March 30, 2010
![Page 32: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/32.jpg)
Cassandra Data Model{ '9xj5ss824mzyv.12345': { 'Record': { 'lat': 40.0149856, 'lon': -105.2705456, 'city': 'Boulder', 'state': 'CO' }, }, 'dr5regy3zcfgr.67890': { 'Record': { 'lat': 40.7142691, 'lon': -74.0059729, 'city': 'New York', 'state': 'NY' } }}
Tuesday, March 30, 2010
![Page 33: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/33.jpg)
Cassandra Data Model{ '9xj5ss824mzyv.12345': { 'Record': { 'lat': 40.0149856, 'lon': -105.2705456, 'city': 'Boulder', 'state': 'CO' }, }, 'dr5regy3zcfgr.67890': { 'Record': { 'lat': 40.7142691, 'lon': -74.0059729, 'city': 'New York', 'state': 'NY' } }}
Tuesday, March 30, 2010
![Page 34: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/34.jpg)
Writes are crazy fastWrites are written to a commit log in the order they’re received - serial I/O
New data is stored in an in-memory table
Memory table is periodically synced to a file
Files are occasionally merged
Reads may end up checking multiple files (bloom filter helps) and merging results• Thats okay because reads are pretty easy to scale
Tuesday, March 30, 2010
![Page 35: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/35.jpg)
How can I query?Depends on the partitioner you use• Random partitioner: makes it really easy to
keep a cluster balanced, but can only do lookups by row key
• Order-preserving partitioner: stores data ordered by row key, so it can query for ranges of keys, but it’s a lot harder to keep balanced
Tuesday, March 30, 2010
![Page 36: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/36.jpg)
BYOI• If you need an index on something other than
the row key, you need to build an inverted index yourself• Row key: attribute you're interested in plus row key
being indexed
• “dr5regy3zcfgr:com.simplegeo/1”
• But what about indexing multiple attributes..?
Tuesday, March 30, 2010
![Page 37: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/37.jpg)
The Curse of DimensionalityLocation data is multidimensional
Traditional GIS software typically uses some variation of a Quadtree or R-Tree for indexes
Like B-Trees, R-Trees need to be updated in-place and are expensive to manipulate when they outgrow memory
Tuesday, March 30, 2010
![Page 38: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/38.jpg)
Dimensionality ReductionIf we think of the world as two-dimensional cartesian plane, we can think of latitude and longitude as coordinates for that plane
Instead of using (x, y) coordinates, we can break the plane into a grid and number each box• Space-filling curve: a continuous line that
intersects every point in a two-dimensional plane
Tuesday, March 30, 2010
![Page 39: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/39.jpg)
Tuesday, March 30, 2010
![Page 40: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/40.jpg)
GeohashA convenient dimensionality reduction mechanism for (latitude, longitude) coordinates that uses a Z-Curve
Simply interleave the bits of a (latitude, longitude) pair and base32 encode the result
Interesting characteristics• Easy to calculate and to reverse
• Represent bounding boxes
• Truncating bits from the end of a geohash results in a larger geohash bounding the original
Tuesday, March 30, 2010
![Page 41: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/41.jpg)
Geohash Drawbacks
Z-Curves are not necessarily the most efficient space-filling curve for range queries• Points on either end of the Z’s diagonal seem
close together when they’re not
• Points next to each other on the spherical earth may end up on opposite sides of our plane
These inefficiencies mean we sometimes have to run multiple queries, or expand bounding box queries to cover very large expanses
Tuesday, March 30, 2010
![Page 42: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/42.jpg)
Geohash AlternativesHilbert curves: improve on Z-Curves but have different drawbacks
Non-algorithmic unique identifiers• Provide unique identifiers for geopolitical and
colloquial bounding polygons
• Yahoo! GeoPlanet’s WOEIDs are a good example
Tuesday, March 30, 2010
![Page 43: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/43.jpg)
Other stuff we use
Tuesday, March 30, 2010
![Page 44: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/44.jpg)
MemcacheUseful for storing ephemeral or short-lived data and for caching
Super crazy extra fast
Robust support from pretty much every language in the world
Tuesday, March 30, 2010
![Page 45: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/45.jpg)
MemcacheDBBDB backed memcache
We use it for statistics• Can’t use Cassandra because it doesn’t
support eventually consistent increment and decrement operations (yet)
Giant con: it’s pretty much impossible to rebalance if you add a node
Tuesday, March 30, 2010
![Page 46: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/46.jpg)
Pushpin ServiceCustom storage solution
R-Tree index for fast lookups
Mostly fixed data sets so it’s ok that we can’t update data efficiently
Tuesday, March 30, 2010
![Page 47: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/47.jpg)
MySQL!
Our website still uses MySQL for some stuff... though we’re moving away from it
Tuesday, March 30, 2010
![Page 48: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/48.jpg)
Thanks!
Tuesday, March 30, 2010
![Page 49: Scaling GIS Data in Non-relational Data Stores](https://reader031.fdocuments.net/reader031/viewer/2022020207/54b7a96e4a7959d3328b45a1/html5/thumbnails/49.jpg)
@mjmalone
Ask me questions!
Tuesday, March 30, 2010