Three Things You Need to Know About Document Data Modeling in NoSQL

Three things you need

to know about data modelingMatthew Revell

[email protected], @matthewrevell

Lead Developer Advocate, Couchbase

1

We’re still learning

Data modeling books

©2014 Couchbase, Inc. 3

Three query methods

Application layer computation Off-load computation

Predictable queries Key-value: pre-computed answers Views

Ad-hoc queries N1QL and key-value with manual

indexes

N1QL and views


Key-value

Key-value

Pre-compute answers asynchronously

Store object state

Choose when to embed data and when to refer to it

Design your keys well


Pre-computed answers

We’re used to asking questions


Key-value look-ups give us a library of answers


Answer-oriented databases


http://martinfowler.com/bliki/AggregateOrientedDatabase.html

Airline bookings


Embed or refer?

How much should we denormalize?

An e-commerce order


The same order in a document


Embedded vs. referred data


You should embed data when:

Speed trumps all else

Slow moving data

No duplication

Application layer can keep multiple copies of same data in sync

When to embed data


You should refer to data when:

Consistency is a priority

The data has large growth potential

When to refer to data


Key design

Key design is as important as document design.

There are three broad types of keys:

Human readable/deterministic: e.g., an email address

Computer generated/random: e.g., a UUID

Compound: e.g., a UUID with a deterministic portion

Three ways to build a key


Human readable/deterministic key


public class user {

private String name;private String email;private String streetAddress;private String city;private String country;private String postCode;private String telephone;private Array orders;private Array productsViewed;

}

{"name": "Matthew Revell","address": "11-21 Paul Street","city": "London","postCode": "EC2A 4JU","telephone": "44-20-3837-9130","orders": [ 1, 9, 698, 32 ],“productsViewed”: [8, 33, 99, 100]

}

Key: [email protected]

Random/computer-generated key


{"name": "Matthew Revell","email": "[email protected]","address": "11-21 Paul Street","city": "London","postCode": "EC2A 4JU","telephone": "44-20-3837-9130","orders": [ 1, 9, 698, 32 ],“productsViewed”: [8, 33, 99, 100]

}

Key: 1001

Using counters to generate keys


Application

user_id = incr(“counter_key")

add(user_id, data)

Creating a new user

add(email_address, user_id)

Application

key = get("[email protected]")

get(key)

Finding a user by email address

mailto:[email protected]

Multiple look-up documents


u::count

1001

u::1001

{ "name": “Matthew Revell",

"facebook_id": 16172910,

"email": “[email protected]”,

“password”: ab02d#Jf02K

"created_at": "5/1/2012 2:30am",

“facebook_access_token”: xox0v2dje20,

“twitter_access_token”: 20jffieieaaixixj }

fb::16172910

1001

nflx::2939202

1001

twtr::2920283830

1001

em::[email protected]

1001

em::[email protected]

1001

uname::mrevell

1001

Compound keys

Compound keys are look-up documents with predictable names.

They continue the discussion of embedded versus referred data.

Compound keys: an example

u::1001

{

"name": "Matthew Revell",

"email": "[email protected]",

"address": "11-21 Paul Street",

"city": "London",

"postCode": "EC2A 4JU",

"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ],

“productsViewed”: [8, 33, 99, 100]

}


u::1001

{




"city": "London",


"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ]

}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}


u::1001

{




"city": "London",


"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ]

}


{"productsList": [

8, 33, 99, 100]

}

p::8

{

id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99,"image": "tshirt.jpg”

}


u::1001

{




"city": "London",


"telephone": "44-20-3837-9130",

"orders": [ 1, 9, 698, 32 ]

}


{"productsList": [

8, 33, 99, 100]

}

p::8

{

id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99

}

p::8::img

“http://someurl.com/tshirt.jpg”

N1QL and Views!

The world beyond key-value

We’re not in Kansas any more


N1QL and views

N1QL Views

Ad-hoc querying Predictable queries

Nested JSON in and unnested JSON

out

Number crunching

Large growth clusters Multidimensional and

geospatial queries


N1QL

Indexes

Schema types and document types

Keyspaces


Indexes


• You always need at least one index

• PRIMARY index gives you all of the keys in the bucket

• Secondary indexes enable ad-hoc querying at speed

Indexes


Indexes


NEW Global Secondary Indexes Views

N1QL query to create a new indexJavaScript map/reduce

Option to run dedicated indexing and

query nodes

Queries always run on database nodes

ForestDB-backed Couchstore-backed

Supports multidimensional and geospatial

queries

Maintaining the schema and working with keyspaces


JOINs work on primary keys and secondary keys.

They JOIN across and within keyspaces (for now, that means buckets).

Airlines:

airline_24 ← Key (“primary key”)

{

"id": "24",

"type": "airline",

"name": "American Airlines",

"iata": "AA",

"icao": "AAL",

"callsign": "AMERICAN",

"country": "United States",

"active": "Y"

}

Routes:

route_5966 ← Key

{

"id": "5966",

"type": "route",

"airline": "AA",

"airlineid": "airline_24", ← This is the foreign key

"sourceairport": "MCO",

"destinationairport": "SEA",

"stops": "0",

"equipment": "737",

"schedule": [...

]

}

Offloading computation to N1QL


N1QL allows you to choose which elements are returned at the top level.

This is incredibly useful because it reduces the need to write complex

parsing logic:

• Within the application

• In client-side, front-end frameworks (Angular/Backbone etc.)

SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport,

r.equipment FROM `travel-sample` r UNNEST r.schedule s JOIN `travel-

sample` a ON KEYS r.airlineid WHERE r.sourceairport='LHR' AND

r.destinationairport='LAX' AND s.day=2 ORDER BY a.name

Questions!

Thanks!

Matthew Revell

[email protected]

@matthewrevell

Three Things You Need to Know About Document Data Modeling in NoSQL

Technology

Transcript of Three Things You Need to Know About Document Data Modeling in NoSQL