Three Things You Need to Know About Document Data Modeling in NoSQL

Post on 14-Aug-2015

370 views 1 download

Tags:

Transcript of Three Things You Need to Know About Document Data Modeling in NoSQL

Three things you need

to know about data modelingMatthew Revell

matthew@couchbase.com, @matthewrevell

Lead Developer Advocate, Couchbase

1

We’re still learning

Data modeling books

©2014 Couchbase, Inc. 3

Three query methods

Application layer computation Off-load computation

Predictable queries Key-value: pre-computed answers Views

Ad-hoc queries N1QL and key-value with manual

indexes

N1QL and views

©2014 Couchbase, Inc. 4

Key-value

Key-value

Pre-compute answers asynchronously

Store object state

Choose when to embed data and when to refer to it

Design your keys well

©2014 Couchbase, Inc. 6

Pre-computed answers

We’re used to asking questions

©2014 Couchbase, Inc. 8

Key-value look-ups give us a library of answers

©2014 Couchbase, Inc. 9

Answer-oriented databases

©2014 Couchbase, Inc. 10

http://martinfowler.com/bliki/AggregateOrientedDatabase.html

Airline bookings

©2014 Couchbase, Inc. 11

Embed or refer?

How much should we denormalize?

An e-commerce order

©2014 Couchbase, Inc. 13

The same order in a document

©2014 Couchbase, Inc. 14

Embedded vs. referred data

©2014 Couchbase, Inc. 15

You should embed data when:

Speed trumps all else

Slow moving data

No duplication

Application layer can keep multiple copies of same data in sync

When to embed data

©2014 Couchbase, Inc. 16

You should refer to data when:

Consistency is a priority

The data has large growth potential

When to refer to data

©2014 Couchbase, Inc. 17

Key design

Key design is as important as document design.

There are three broad types of keys:

Human readable/deterministic: e.g., an email address

Computer generated/random: e.g., a UUID

Compound: e.g., a UUID with a deterministic portion

Three ways to build a key

©2014 Couchbase, Inc. 19

Human readable/deterministic key

©2014 Couchbase, Inc. 20

public class user {

private String name;private String email;private String streetAddress;private String city;private String country;private String postCode;private String telephone;private Array orders;private Array productsViewed;

}

{"name": "Matthew Revell","address": "11-21 Paul Street","city": "London","postCode": "EC2A 4JU","telephone": "44-20-3837-9130","orders": [ 1, 9, 698, 32 ],“productsViewed”: [8, 33, 99, 100]

}

Key: matthew@couchbase.com

Random/computer-generated key

©2014 Couchbase, Inc. 21

{"name": "Matthew Revell","email": "matthew@couchbase.com","address": "11-21 Paul Street","city": "London","postCode": "EC2A 4JU","telephone": "44-20-3837-9130","orders": [ 1, 9, 698, 32 ],“productsViewed”: [8, 33, 99, 100]

}

Key: 1001

Using counters to generate keys

©2014 Couchbase, Inc. 22

Application

user_id = incr(“counter_key")

add(user_id, data)

Creating a new user

add(email_address, user_id)

Application

key = get("matthew@couchbase.com")

get(key)

Finding a user by email address

Multiple look-up documents

©2014 Couchbase, Inc. 23

u::count

1001

u::1001

{ "name": “Matthew Revell",

"facebook_id": 16172910,

"email": “matthew@couchbase.com”,

“password”: ab02d#Jf02K

"created_at": "5/1/2012 2:30am",

“facebook_access_token”: xox0v2dje20,

“twitter_access_token”: 20jffieieaaixixj }

fb::16172910

1001

nflx::2939202

1001

twtr::2920283830

1001

em::matthew@couchbase.com

1001

em::matthew@understated.co.uk

1001

uname::mrevell

1001

Compound keys

Compound keys are look-up documents with predictable names.

They continue the discussion of embedded versus referred data.

Compound keys: an example

u::1001

{

"name": "Matthew Revell",

"email": "matthew@couchbase.com",

"address": "11-21 Paul Street",

"city": "London",

"postCode": "EC2A 4JU",

"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ],

“productsViewed”: [8, 33, 99, 100]

}

Compound keys: an example

u::1001

{

"name": "Matthew Revell",

"email": "matthew@couchbase.com",

"address": "11-21 Paul Street",

"city": "London",

"postCode": "EC2A 4JU",

"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ]

}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}

Compound keys: an example

u::1001

{

"name": "Matthew Revell",

"email": "matthew@couchbase.com",

"address": "11-21 Paul Street",

"city": "London",

"postCode": "EC2A 4JU",

"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ]

}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}

p::8

{

id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99,"image": "tshirt.jpg”

}

Compound keys: an example

u::1001

{

"name": "Matthew Revell",

"email": "matthew@couchbase.com",

"address": "11-21 Paul Street",

"city": "London",

"postCode": "EC2A 4JU",

"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ]

}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}

p::8

{

id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99

}

p::8::img

“http://someurl.com/tshirt.jpg”

N1QL and Views!

The world beyond key-value

We’re not in Kansas any more

©2014 Couchbase, Inc. 30

N1QL and views

N1QL Views

Ad-hoc querying Predictable queries

Nested JSON in and unnested JSON

out

Number crunching

Large growth clusters Multidimensional and

geospatial queries

©2014 Couchbase, Inc. 31

N1QL

Indexes

Schema types and document types

Keyspaces

©2014 Couchbase, Inc. 32

Indexes

©2014 Couchbase, Inc. 33

• You always need at least one index

• PRIMARY index gives you all of the keys in the bucket

• Secondary indexes enable ad-hoc querying at speed

Indexes

©2014 Couchbase, Inc. 34

Indexes

©2014 Couchbase, Inc. 35

NEW Global Secondary Indexes Views

N1QL query to create a new indexJavaScript map/reduce

Option to run dedicated indexing and

query nodes

Queries always run on database nodes

ForestDB-backed Couchstore-backed

Supports multidimensional and geospatial

queries

Maintaining the schema and working with keyspaces

©2014 Couchbase, Inc. 36

JOINs work on primary keys and secondary keys.

They JOIN across and within keyspaces (for now, that means buckets).

Airlines:

airline_24 ← Key (“primary key”)

{

"id": "24",

"type": "airline",

"name": "American Airlines",

"iata": "AA",

"icao": "AAL",

"callsign": "AMERICAN",

"country": "United States",

"active": "Y"

}

Routes:

route_5966 ← Key

{

"id": "5966",

"type": "route",

"airline": "AA",

"airlineid": "airline_24", ← This is the foreign key

"sourceairport": "MCO",

"destinationairport": "SEA",

"stops": "0",

"equipment": "737",

"schedule": [...

]

}

Offloading computation to N1QL

©2014 Couchbase, Inc. 37

N1QL allows you to choose which elements are returned at the top level.

This is incredibly useful because it reduces the need to write complex

parsing logic:

• Within the application

• In client-side, front-end frameworks (Angular/Backbone etc.)

SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport,

r.equipment FROM `travel-sample` r UNNEST r.schedule s JOIN `travel-

sample` a ON KEYS r.airlineid WHERE r.sourceairport='LHR' AND

r.destinationairport='LAX' AND s.day=2 ORDER BY a.name

Questions!

Thanks!

Matthew Revell

matthew@couchbase.com

@matthewrevell