Is NoSQL the Future of Data Storage? By Gary Short Developer Express.
-
date post
20-Dec-2015 -
Category
Documents
-
view
223 -
download
0
Transcript of Is NoSQL the Future of Data Storage? By Gary Short Developer Express.
![Page 1: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/1.jpg)
Is NoSQL the Future of Data Storage?
By Gary ShortDeveloper Express
![Page 2: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/2.jpg)
Introduction
• Gary Short• Technical Evangelist for Developer Express• C# MVP• [email protected]• www.garyshort.org• @garyshort.
![Page 3: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/3.jpg)
![Page 4: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/4.jpg)
Where Does NoSQL Originate?
• 1998– OS relational database• Didn’t expose an SQL interface
• Created by Carlo Strozzi– Said the NoSQL movement• “departs from the relational model altogether...”• “...should have been called ‘NoREL”.
![Page 5: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/5.jpg)
More Recently...
• Eric Evans reintroduced the term in 2009– Johan Oskarsson (last.fm)• Event to discuss OS distributed databases
• This labels growing number datastores– Open source– Non-relational– Distributed– (often) don’t guarantee ACID.
![Page 6: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/6.jpg)
Atlanta 2009
• No:sql(east) conference• Billed as “conference of no-rel datastores”• Worst tag line ever– SELECT fun, profit FROM real_world WHERE rel=false.
![Page 7: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/7.jpg)
Not Ant-RDBMS
![Page 8: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/8.jpg)
Key Attributes of NoSQL Databases
• Don’t require fixed table schemas• Non-relational• (Usually) avoid join operations• Scale horizontally– Adding more nodes to a storage system.
![Page 9: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/9.jpg)
What Does the Taxonomy Look Like?
![Page 10: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/10.jpg)
Document Store
• Apache Jackrabbit• CouchDB• MongoDB• SimpleDB• XML Databases– MarkLogic Server– eXist.
![Page 11: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/11.jpg)
Document What?
• Okay think of a web page...– Relational model requires column/tag– Lots of empty columns– Wasted space
• Document model just stores the pages as is– Saves on space– Very flexible.
![Page 12: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/12.jpg)
Graph Storage
• AllegroGraph• Core Data• Neo4j• DEX• FlockDB.
![Page 13: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/13.jpg)
Which Means?
• Graph consists of– Node (‘stations’ of the graph)– Edges (lines between them)
• FlockDB– Created by the Twitter folks– Nodes = Users– Edges = Nature of relationship between nodes.
![Page 14: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/14.jpg)
Key/Value Stores
• On disk• Cache in Ram• Eventually Consistent
– Weak Definition• “If no updates occur for a period, eventually all updates will
propagate through the system and all replicas will be consistent”
– Strong Definition• “for a given update and a given replica eventually either the
update reaches the replica or the replica retires”
• Ordered– Distributed Hash Table allows lexicographical processing.
![Page 15: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/15.jpg)
Object Databases
• Db4o• GemStone/S• InterSystems Caché• Objectivity/DB• ZODB.
![Page 16: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/16.jpg)
Okay got it, Now Let’s Compare Some Real World Scenarios
![Page 17: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/17.jpg)
You Need Constant Consistency
• You’re dealing with financial transactions• You’re dealing with medical records• You’re dealing with bonded goods• Best you use a RDMBS .
![Page 18: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/18.jpg)
You Need Horizontal Scalability
• You’re working across defined timezones• You’re Aggregating large quantities of data• Maintaining a chat server (Facebook chat)• Use NoSQL.
![Page 19: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/19.jpg)
Up in the Clouds Baby
• If you are using Azure or AWS– Compare costs of Azure Storage or SimpleDB to
SQL Azure or Elastic RDBMS• Could be cheaper for your scenario.
![Page 20: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/20.jpg)
It’s all About the iPhone!
![Page 21: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/21.jpg)
Frequently Written Rarely Read
• Think web counters and the like• Every time a user comes to a page = ctr++• But it’s only read when the report is run• Use NoSQL (key-value storage).
![Page 22: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/22.jpg)
I Got Big Data!
• Think weather stats• Satellite Images• Maps• Use NoSQL ( Something like Hadoop).
![Page 23: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/23.jpg)
Binary Baby!
• If you are YouTube• Flickr• Twitpic• Spotify• NoSQL (Amazon S3).
![Page 24: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/24.jpg)
Here Today Gone Tomorrow
• Transient data like..– Web Sessions– Locks– Short Term Stats• Shopping cart contents
• Use NoSQL (Memcache).
![Page 25: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/25.jpg)
Data Replication
• Same data in two or more locations– Music Library• Web browser• iPone App
• NoSQL (CouchDB).
![Page 26: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/26.jpg)
Hit me Baby One More Time!
• High Availability– High number of important transactions• Online gambling• Pay Per view
– Ahem!
• Online Auction
• NoSQL (Cassandra – automatic clustering).
![Page 27: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/27.jpg)
Give me a Real World Example
• Twitter– The challenges• Needs to store many graphs
– Who you are following– Who’s following you– Who you receive phone notifications from etc
• To deliver a tweet requires rapid paging of followers• Heavy write load as followers are added and removed• Set arithmetic for @mentions (intersection of users).
![Page 28: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/28.jpg)
What Did They Try?
• Relational Databases• Key-Value storage of denormalized lists• Did it work?– Nope!• Either good at
– Handling the write load– Or paging large amounts of data– But not both .
![Page 29: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/29.jpg)
What Did They Need?
• Simplest possible thing that would work• Allow for horizontal partitioning• Allow write operations to– Arrive out of order– Or be processed more than once
• Failures should result in redundant work– Not lost work!
![Page 30: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/30.jpg)
The Result was FlockDB
• Stores graph data• Not optimised for graph traversal operations• Optimised for large adjacency lists– List of all edges in a graph• Key is the edge value a set of the node end points
• Optimised for fast read and write• Optimised for page-able set arithmetic.
![Page 31: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/31.jpg)
How Does it Work?
• Stores graphs as sets of edges between nodes• Data is partitioned by node– All queries can be answered by a single partition
• Write operations are idempotent– Can be applied multiple times without changing
the result• And commutative– Changing the order of operands doesn’t change
the result.
![Page 32: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/32.jpg)
Commutative Writes Help Bring up Partitions
• Partition can receive write traffic immediately• Receive dump of data in the background• Live for read as soon as the dump is complete.
![Page 33: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/33.jpg)
Performance?
• Currently store 13 billion edges• 20K writes / second• 100K reads / second.
![Page 34: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/34.jpg)
Lessons Learned?
• Use aggressive timeouts– Cut a client loose after timeout expired– Let it try again on another app server
• Use same code path for error and normal ops– Error requests are periodically retried
• Instrument.
![Page 35: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/35.jpg)
Punchline?
• Under all the bells and whistles...– Its MySQL .
![Page 36: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/36.jpg)
So is this the Future?
• Yes!• And No!
![Page 38: Is NoSQL the Future of Data Storage? By Gary Short Developer Express.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d415503460f94a1b25b/html5/thumbnails/38.jpg)
Coming up…P/X001Understanding and Preventing SQL Injection AttacksKevin KlineP/L001SSIS FieldnotesDarren GreenP/L002The (Geospatial) Shapes of Things to ComeSimon MunroP/L005End to End Master Data Management with SQL Server Master Data ServicesJeremy KashelP/T007Understanding Microsoft Certification in SQL ServerChris Testa-O'Neill
#SQLBITS