Patterns / Antipatterns with NoSQL
-
Upload
luca-bonmassar -
Category
Technology
-
view
7.102 -
download
1
description
Transcript of Patterns / Antipatterns with NoSQL
![Page 1: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/1.jpg)
Luca Bonmassar
[Anti]Patterns with NoSQL
![Page 2: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/2.jpg)
NOSQL DAY ’11
NoSQL vs. SQL
![Page 3: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/3.jpg)
NOSQL DAY ’11
NoSQL / NoAgenda
No specific NoSQL technologies (document, key/value, ...)
No specific NoSQL db features (sharding, replication, ...)
No specific ...
...
![Page 4: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/4.jpg)
NOSQL DAY ’11
Does it solve my problem?
![Page 5: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/5.jpg)
NOSQL DAY ’11
Does it solve my problem?
![Page 6: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/6.jpg)
NOSQL DAY ’11
Does it solve my problem?
But what is your problem?
![Page 7: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/7.jpg)
NOSQL DAY ’11
Does it solve my problem?
But what is your problem?
NoSQL fragmentation (like NoCloud, NoJava, NoMicrosoft, ...)
![Page 8: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/8.jpg)
NOSQL DAY ’11
[Anti]PatternsThe “dynamic” website
Data mapping
The Alien
The binary Alien
The SQLQueue
Overnormalized
Unpredictable tomorrow
![Page 9: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/9.jpg)
NOSQL DAY ’11
Who am I?
Passionate about technology
Favorite topics: Cloud, Virtualization, NoSQL
Gamification fan! (or fun?)
![Page 10: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/10.jpg)
The “dynamic” website
![Page 11: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/11.jpg)
NOSQL DAY ’11
The “dynamic” website
![Page 12: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/12.jpg)
NOSQL DAY ’11
The “dynamic” website
I have a website
![Page 13: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/13.jpg)
NOSQL DAY ’11
The “dynamic” website
I have a website
Dynamic contents everywhere
![Page 14: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/14.jpg)
NOSQL DAY ’11
The “dynamic” website
I have a website
Dynamic contents everywhere
High DB load
![Page 15: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/15.jpg)
NOSQL DAY ’11
The “dynamic” website
![Page 16: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/16.jpg)
NOSQL DAY ’11
The “dynamic” website
![Page 17: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/17.jpg)
NOSQL DAY ’11
The “dynamic” website
SELECT COUNT(###) WHERE ...
SELECT COUNT(###) WHERE ...
SELECT COUNT(###) WHERE ...
SELECT username FROM users WHERE
sessionid = ###
SELECT avatar.* FROM users WHERE ...
SELECT users.*, count(distinct puzzle_id) FROM submissions JOIN users on submissions.user_id = users.id where submissions.current_state =
"succeeded" ...
SELECT COUNT("submissions"."puzzle_id") FROM "submissions" WHERE (puzzle_id IS
NOT NULL) AND (user_id IS NOT NULL) GROUP BY puzzle_id ...
SELECT username FROM users WHERE
sessionid = ###
SELECT * FROM submissions WHERE ...
![Page 18: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/18.jpg)
NOSQL DAY ’11
The “dynamic” website
SELECT COUNT(###) WHERE ...
SELECT COUNT(###) WHERE ...
SELECT COUNT(###) WHERE ...
SELECT username FROM users WHERE
sessionid = ###
SELECT avatar.* FROM users WHERE ...
SELECT users.*, count(distinct puzzle_id) FROM submissions JOIN users on submissions.user_id = users.id where submissions.current_state =
"succeeded" ...
SELECT COUNT("submissions"."puzzle_id") FROM "submissions" WHERE (puzzle_id IS
NOT NULL) AND (user_id IS NOT NULL) GROUP BY puzzle_id ...
SELECT username FROM users WHERE
sessionid = ###
SELECT * FROM submissions WHERE ...
![Page 19: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/19.jpg)
NOSQL DAY ’11
The “dynamic” website
![Page 20: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/20.jpg)
NOSQL DAY ’11
The “dynamic” website
SELECT COUNT(###) WHERE ...
SELECT COUNT(###) WHERE ...
SELECT COUNT(###) WHERE ...
SELECT username FROM users WHERE
sessionid = ###
SELECT avatar.* FROM users WHERE ...
SELECT users.*, count(distinct puzzle_id) FROM submissions JOIN users on submissions.user_id = users.id where submissions.current_state =
"succeeded" ...
SELECT COUNT("submissions"."puzzle_id") FROM "submissions" WHERE (puzzle_id IS
NOT NULL) AND (user_id IS NOT NULL) GROUP BY puzzle_id ...
SELECT username FROM users WHERE
sessionid = ###
SELECT * FROM submissions WHERE ...
![Page 21: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/21.jpg)
NOSQL DAY ’11
The “dynamic” website
Is it that dynamic?
Are there any data structures that fit better than relational?
![Page 22: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/22.jpg)
NOSQL DAY ’11
Cache, Cache, Cache
Use Ad-Hoc Caching (e.g. Rails Caching)
SQL Table as cache
![Page 23: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/23.jpg)
NOSQL DAY ’11
SQL Table as Cache
When you use SQL Table as Cache, a kitten somewhere dies
Use Memcache, MemcacheDB, Redis
![Page 24: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/24.jpg)
NOSQL DAY ’11
Better tools?
![Page 25: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/25.jpg)
NOSQL DAY ’11
Rethink data structures
SELECT COUNT(*) ...SELECT COUNT(*) ...SELECT COUNT(*) ...
submitted++passed++failed++
![Page 26: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/26.jpg)
NOSQL DAY ’11
Rethink data structures
SELECT *, count(distinct ...) FROM submissions JOIN users on ... where submissions.current_state = "succeeded" ...
List(5)List.add / List.replace
![Page 27: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/27.jpg)
Data mapping
![Page 28: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/28.jpg)
NOSQL DAY ’11
Data mapping
An original idea: we need a newsfeed!
Users can comment on feed items
Users can reply to comments
![Page 29: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/29.jpg)
NOSQL DAY ’11
What the customer is looking for
![Page 30: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/30.jpg)
NOSQL DAY ’11
What the developer is thinking about
![Page 31: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/31.jpg)
But how to ...
![Page 32: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/32.jpg)
NOSQL DAY ’11
Eureka!
The item has a feed_id and a parent_id
You have now a navigable SQL data structure!
![Page 33: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/33.jpg)
NOSQL DAY ’11
Data mapping - alternatives
Do not try this at home! (or any office)
A document db can help
{ feed : 42, user : ‘luca.bonmassar’, data : ‘living the dream!’, comments : [{user : ‘jonny’ : ‘make sense!’ }] }
![Page 34: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/34.jpg)
The Alien
![Page 35: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/35.jpg)
NOSQL DAY ’11
The Alien
A type of object stored in a relational database
No (or weak) relations with any other table
Stored in the db because the db == persistency
![Page 36: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/36.jpg)
NOSQL DAY ’11
User Settings
User preferences
user_id, data
Data as BLOB or TEXT
![Page 37: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/37.jpg)
NOSQL DAY ’11
The Alien
The Alien is a key/value entity
Use a key/value storage to store it
![Page 38: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/38.jpg)
The binary Alien
![Page 39: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/39.jpg)
NOSQL DAY ’11
The binary Alien
Like the Alien, but the payload is pure binary data
Common solution to store “small” images
Even worse: Base64 encoded binary
![Page 40: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/40.jpg)
NOSQL DAY ’11
The binary Alien
Move the binary Alien to a binary content provider
S3 or Filesystem are good ones
Let the webserver access/serve them directly
![Page 41: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/41.jpg)
The SQLQueue
![Page 42: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/42.jpg)
NOSQL DAY ’11
The SQLQueue
Distributed components need to exchange data
Producers / Consumers
Backlog of work to be completed
SQL database (== persistency) as persistent queue
![Page 43: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/43.jpg)
NOSQL DAY ’11
Mail delivery
![Page 44: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/44.jpg)
NOSQL DAY ’11
Mail delivery
![Page 45: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/45.jpg)
NOSQL DAY ’11
Mail delivery
![Page 46: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/46.jpg)
NOSQL DAY ’11
Mail delivery
![Page 47: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/47.jpg)
NOSQL DAY ’11
The SQLQueue
No “relations”, like the Alien
Simulating a Queue using AUTO_INCREMENT ids and transactions
![Page 48: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/48.jpg)
NOSQL DAY ’11
SQLQueue
In some countries, SQLQueue is considered a crime against software
Use message queues(Amazon SQS, MemcacheQ, StormMQ, RabbitMQ, ... )
![Page 49: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/49.jpg)
Overnormalized
![Page 50: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/50.jpg)
NOSQL DAY ’11
Overnormalized
![Page 51: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/51.jpg)
NOSQL DAY ’11
Overnormalized
The process of organizing data to minimize redundancy
![Page 52: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/52.jpg)
NOSQL DAY ’11
Overnormalized
The process of organizing data to minimize redundancy
A larger schema is broken into smaller ones
![Page 53: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/53.jpg)
NOSQL DAY ’11
Overnormalized
The process of organizing data to minimize redundancy
A larger schema is broken into smaller ones
user_id, email
![Page 54: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/54.jpg)
NOSQL DAY ’11
Overnormalized
The process of organizing data to minimize redundancy
A larger schema is broken into smaller ones
user_id, email
user_id, phone_num
![Page 55: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/55.jpg)
NOSQL DAY ’11
Overnormalized
The process of organizing data to minimize redundancy
A larger schema is broken into smaller ones
user_id, email
user_id, phone_num
user_id, badge
![Page 56: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/56.jpg)
NOSQL DAY ’11
Overnormalized
PRO:
reduce redundancy
less overhead
each table scale separately
![Page 57: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/57.jpg)
NOSQL DAY ’11
OvernormalizedCons:
![Page 58: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/58.jpg)
NOSQL DAY ’11
OvernormalizedCons:
... JOIN... JOIN... JOIN... JOIN... JOIN... JOIN... JOIN...
![Page 59: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/59.jpg)
NOSQL DAY ’11
Overnormalized
Cache the normalized data
Denormalize / keep a replica of the denormalized view
Use document db or key/value storage for the replica
![Page 60: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/60.jpg)
Unpredictable tomorrow
![Page 61: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/61.jpg)
NOSQL DAY ’11
Unpredictable tomorrow
You are now part of a new Agile(TM) project
You are Agile(TM), so:
No complete specs
No complete use cases
![Page 62: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/62.jpg)
NOSQL DAY ’11
How many wonderful things around me
A mobile App w/ internet backend
“Simple” use cases
User can log in
User can update their location
User can get all the many wonderful things they have around themselves
![Page 63: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/63.jpg)
NOSQL DAY ’11
How many wonderful things around me
Data Model:
User
Places
Aliens!!! Aliens!!! Aliens!!!
![Page 64: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/64.jpg)
NOSQL DAY ’11
...but sooner or later...
![Page 65: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/65.jpg)
NOSQL DAY ’11
How many wonderful things around me
Users can send/receive friendship invitations
Users can import FB friends, Twitter followers
Users can interact with messages, pokes
Users can check-in into places
Users can share their checkins with friends
...
![Page 66: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/66.jpg)
NOSQL DAY ’11
Unpredictable tomorrow
No silver bullets
Mix technologies
E.g. relational databases to handle relations
Migrations are painful, but always an option
![Page 67: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/67.jpg)
One more thing
![Page 68: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/68.jpg)
NOSQL DAY ’11
Recap
Does it make sense with the relational paradigm?
Do I need a persistent storage or a relational database?
![Page 69: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/69.jpg)
NOSQL DAY ’11
Thanks! Any Questions?
![Page 70: Patterns / Antipatterns with NoSQL](https://reader034.fdocuments.net/reader034/viewer/2022052303/554d0165b4c905d4568b461a/html5/thumbnails/70.jpg)
NOSQL DAY ’11
Contacts
linkedin.com/in/lucabonmassar
twitter.com/lucabonmassar
joind.in/talk/view/2939