NoSQL at Scale: Architecture and Common Pitfalls

Arthur Gimpel DataZone

About Me

• Working with databases for the past 8 years

• 5 years as SQL Server DBA & .NET

• 3 years with NoSQL, Python, Node.js

• 2015 - DataZone

DataZone Data is our business

• High end consultancy & projects

• Private and public training

• Multi vendor MSP (Managed Service Provider)

• Part of Matrix, CloudZone Family

• Proud partners of:

NoSQL Why not?

All the wrong reasons..

• Web scale! web scale!

• Cost reduction

• Free school for developers

Common Pitfalls & Best Practices

Normalizing in the low cost storage era

Expectations gone wild• Stored procedures

• Jobs

• Triggers

• Mailing Servers

• Shell Commands

ODM != ORMORM

• Close the modelling gaps

• Simple Standard CRUD

ODM

• Typing the untyped

• Retries

• Transactions

• Events

Best Practices• DynamoDB

• Streams: Change Data Capture

• Cross region replication & synchronisation

• Event driven development: Send your new user a welcome email

• Polyglot Persistence: Initiate an ETL, update your Elasticsearch index

Best Practices• Elasticsearch

• AWS Plugin:

• Allow service discovery based on security groups

• Run snapshots & restore directly to/from s3

• Storage:

• In small clusters, use EBS.

• In large clusters, use instance store, if you have replication enabled.

• Leverage Watcher & AWS Lambda for anything!

Best Practices• Couchbase

• XDCR: • Make sure you use cross datacenter replication for multi

region deployments

• Use encrypted communication, especially on XDCR

• Sizing: • All members must be identical

• Prefer Memory & IO Optimized for most deplyments

When Family Can Be Chosen

• Every engine has its flavours - know your tradeoffs

• Splitting drives is good for on prem. EBS is different

• Scaling up & down is always X2

• Prefer linux AMI by AWS- always check for official support

Eventually…• Don’t work for technology. Make technology for

you. Chose your engines wisely

• Beware of your cluster’s distribution model - don’t over-estimate elasticity

• Experimenting is easy: Change, Benchmark, Analyse. Repeat.

• Run engine backups - not EBS snapshots

Thank You!

Arthur Gimpel arthurgi@datazone.io