NoSQL Databases for Implementing Data Services – Should I Care?

2012 © Trivadis

NoSQL Databases for Implementing Data Services – Should I Care?1

BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

Welcome NoSQL Databases for Implementing Data Services – Should I Care ?Guido Schmutz

UKOUG Conference 2012

04.12.2012

10.10.2012

2012 © Trivadis


Guido Schmutz

• Working for Trivadis for more than 15 years

• Oracle ACE Director for Fusion Middleware and SOA

• Co-Author of different books

• Consultant, Trainer Software Architect for Java, Oracle, SOA and EDA

• Member of Trivadis Architecture Board

• Technology Manager @ Trivadis

• More than 20 years of software development experience

• Contact: [email protected]

• Blog: http://guidoschmutz.wordpress.com

• Twitter: gschmutz

10.10.2012

mailto:[email protected]

http://guidoschmutz.wordpress.com/

http://twitter.com/

http://entwickler-press.de/ep/ep_buecher/pspic/cover_original/94/39390842394488241c02194.jpg

2012 © Trivadis


Agenda

1. Why NoSQL and what is it?

2. NoSQL Database Types

3. Polyglot Persistence

4. NoSQL and Service-Oriented Architecture

5. Summary

10.10.2012

2012 © Trivadis


History of Database

1960s File-based, Network (CODASYL) and Hierarchical Databases

1970s Relational Database

1980 SQL became the standard query language

Early 1990 Object-Databases

Late 1990 XML Databases

2004 NoSQL Databases

10.10.2012

2012 © Trivadis


What‘s wrong with Relational Databases ? They are great ….

• SQL provides a rich, declarative query language

• Database enforce referential integrity

• ACID semantics

• Well understood by developers, database administrators

• Well supported by different languages, frameworks and tools• Hibernate, JPA, JDBC, iBATIS, Entity Framework

• Well understood and accepted by operations people (DBAs)• Configuration• Monitoring• Backup and Recovery• Tuning• Design

10.10.2012

2012 © Trivadis


Relational Databases are great ... But!

Problem: Complex Object graphs Object/Relational impedance mismatch Complicated to map rich domain model

to relational schema Performance issues

Many rows in many tables Many joins Eager vs. lazy loading

Problem: Schema evolution Adding attributes to an object => have to add columns to table Expensive, if lots of data in that table

- Holding locks on the tables for long time- Application downtime …

10.10.2012

ORDER

ADDRESS

CUSTOMER

ORDER_LINES

Order

ID: 1001Order Date: 15.9.2012

Line Items

Customer

First Name: PeterLast Name: Sample

Billing Address

Street: Somestreet 10City: SomewherePostal Code: 55901

Name

Ipod Touch

Monster Beat

Apple Mouse

Quantity

1

2

1

Price

220.95

190.00

69.90

2012 © Trivadis



10.10.2012

O/R Mapping

RDBMS

Repository/DAO

Service

Consumer

REST/SOAP

SQL

ORDER

ADDRESS

CUSTOMER

ORDER_LINES

Order

ID: 1001Order Date: 15.9.2012

Line Items

Customer


Billing Address


Name

Ipod Touch

Monster Beat

Apple Mouse

Quantity

1

2

1

Price

220.95

190.00

69.90

2012 © Trivadis



Problem: Semi-structured data Relational schema doesn‘t easily handle semi-structured data Common solutions

- Name/Value table- Poor performance- Lack of constraint

- Serialize as Blob- Fewer joins, but no query capabilities

Problem: Scaling Scaling writes difficult/expensive/impossible => BigData Vertical scaling is limited and is expensive Horizontal scaling is limited and is expensive

10.10.2012

2012 © Trivadis


Solution: NoSQL ?

No standard definition of what NoSQL means

• Not Only SQL

Term began in a workshop organized in 2009

but some common characteristics of NoSQL databases

• They don‘t use the relational data model and thus don‘t use SQL

• Tend to be designed to run on cluster

• Tend to be Open Source

• Schema-Less - Don‘t have a fixedschema, allowing to store any data in any record

• Different APIs

10.10.2012

Search

Transactions

Caching

Blobs

Batch

Triggers

Object-Relational Relational-Object

User Interface

Caching

Transactions

Search

Batch

Data

User Interface

Key Value Stores

Lucene

MapReduce Ser

vic

es

RDBMS NoSQL

Middle Tier

Database Tier

Presentation Tier

2012 © Trivadis


Central vs. Application Databases

10.10.2012

Application Database

• Only accessed by a single application

• Only the application using the database needs to know about the structure

• Easier to maintain and evolve the schema

• More freedom to choose the database

• Applicable to SOA (i.e. Data Service/Entity Service) with good Service Autonomy

• Ready for the cloud

Central Database

• Using SQL as the integration mechanism between applications

• applications store data in common DB

• Improves communication, all applications operate on consistent set of data

• Structure ends up to be more complex

• Changes need to be coordinated with all other applications using the database

• Side-effects (i.e. adding database index)

DB

Application 1 Application 2 Application 3

DB

Application 1 Application 2 Application 3

DB DB

2012 © Trivadis


Relational vs. Aggregate Data Models

10.10.2012

Aggregate is a term that comes from Domain-Driven Design (Evans)

An aggregate is a collection of related objects, that should be treated as a unit Unit for data manipulation and

management of consistency

The relational model takes the information and divides it into tuples (rows)

A tuple is a limited data structure no nesting of tuples no list of values

2012 © Trivadis


Relational vs. Aggregate Data Model

10.10.2012

Aggregate InstanceRelational Instance

ID NAME

CUSTOMER

1 Guido

ID CUSTOMER_ID

BILLING_ADDRESS

1 1

ADDRESS_ID

55

ID STREET

ADDRESS

55 Chaumontweg

CITY

Spiegel

POST_CODE

3095

ID CUSTOMER_ID

ORDER

90 1

SHIPPING_ADDRESS_ID

55

{„id“:1,„name“:“Guido“,„billingAddress“:[{„street“:“Chaumontweg“,“city“:“Spiegel“,“postCode“:“3095“}]}

{„id“:90,„customerId“:1,„orderItems“:[{„productId“:1000,“price“: 250.55, „produtName“: „iPod Touch“},{„productId“:1020,“price“: 199.55, „produtName“: „Monster Beat“}],

„sippingAddress“:[{„street“:“Chaumontweg“,“city“:“Spiegel“,“postCode“:“3095“}]}

ID ORDER_ID

ORDER_ITEM

1 90

PRODUCT_ID

1000

PRICE

250.55

ID NAME

PRODUCT

1000 IPod Touch

1020 Monster Beat

1 90 1020 199.55

2012 © Trivadis


Brewer‘s CAP Theorem

A distributed system can support only two of the following characteristics

• ConsistencyAll of the nodes see the same data at the same time, regardless of where the data is stored

• AvailabilityNode failures do not preventsurvivors from continuing to operate

• Partition toleranceThe system continues to operate despite arbitrary message loss

10.10.2012

Source: Making Sense of NoSQL – Manning Publications

2012 © Trivadis


ACID vs. BASE style Transactions

10.10.2012

Basically Available

Soft State

Eventually Consistent

Weak consistency – stale data OK

Availability first

Best Effort

Approximate answers OK

Aggressive (optimistic)

Simpler and Faster

Atomic

• All of the work in a transaction completes (commit) or none of it

Consistent

• Transaction transforms the database from one consistent state to another consistent state

Isolated

• results of any changes made during a transaction are not visible until committed

Durable

• Results of committed transaction survive failures

2012 © Trivadis


Agenda





5. Summary

10.10.2012

2012 © Trivadis

NoSQL Database Types

Key/Value Stores

Ordered Key-Value Stores

Big Table Stores (map-of-maps-of-maps)

Document Stores

Graph Databases

10.10.2012NoSQL Databases for Implementing Data Services – Should I Care?

16

Key/Value Column Family Document Graph

Design Collections of Key/Value Pairs

Columns and Colum Families. Accesses directly the column values.

Key/Value pairs but value is interpreted by the database

Focus on the connections between data and the fast navigation

Scalability/Performance

+++ +++ ++ ++

Aggregate-oriented

Yes Yes Yes No

Complexity + ++ ++ +++

Inspiration and Relation

Berkley DB,Memcached,Distributed Hashmaps

SAP Sybase IQ, BigTable

Lotus Notes Graph Theory

NoSQL Products

VoldemortRedisRiak

HbaseCassandraHypertableAmazon SimpleDB

CouchDBMongoDBOrientDBRavenDB

SonesNeo4JInfoGridFlockDB

2012 © Trivadis


NoSQL Database Types

10.10.2012

Siz

e

Complexity

Key-value stores

Column Family

Document

Graph

Relational

2012 © Trivadis


Key Value Databases

A key-value store is a simple hash table

Primarily used when all access to the database is via primary key

Simplest NoSQL data stores to use (from an API perspective)

PUT, POST, GET, DELETE (matches REST)

Value is a blob with the data store not caring or knowing what is inside

Aggregate-Oriented

10.10.2012

Suitable Use Cases

• Storing Session Information

• User Profiles, Preferences

• Shopping Cart Data

2012 © Trivadis


Column-Family Stores

Store data in column families as rows that have many columns associated with a row key

Column families are groups of related data, often accessed together

Aggregate-Oriented

10.10.2012

Suitable Use Cases

• Event Logging

• Content Management Systems

• Counters

• Expiring UsageSource: NoSQL Distilled

2012 © Trivadis


Document Databases

Documents are the main concept

Stores and retrieves documents, which can be XML, JSON, BSON, …

Documents are self-describing, hierarchical tree data structures which can consist of maps, collections and scalar values

Documents stored are similar to each other but do not have to be exactly the same

Aggregate-Oriented

10.10.2012

Suitable Use Cases

• Event Logging

• Content Management Systems

• Web Analytics or Real-Time Analytics

• Product Catalog

2012 © Trivadis


Document Database - MongoDB

10.10.2012

2012 © Trivadis


Graph Databases

Allow to store entities and relationships between these entities

Entities are known as nodes, which have properties

Relations are known as edges, which also have properties

A query on the graph is also known as traversing the graph

Traversing the relationships is very fast

10.10.2012

Suitable Use Cases

• Connected Data

• Routing, Dispatch and Location-Based Services

• Recommendation Engines

Customer

Address

Country

ADDRESS

COUNTRY

Order

BILLING_ADDRESS

DELIVERY_ADDRESS

Product

Tag

RATED TAG

LINE_ITEM

2012 © Trivadis


Graph Database – Neo4J

10.10.2012

START MATCH WHERE RETURN ORDER BY LIMIT

customer=node:Customer(email = "[email protected]")

customer-[:ORDERED]->order-[item:LINEITEM]->product

order.date > 20120101

product.name, sum(item.amount) AS product

products DESC 20

Query through Cypher

2012 © Trivadis


Agenda





5. Summary

10.10.2012

2012 © Trivadis


Polyglot Persistence

In 2006, Neal Ford coined the term Polyglot Programming

Applications should be written in a mix of languages to take advantage of the fact that different languages are suitable for tackling different problems

Polyglot Persistence defines a a hybrid approach to persistence

Using multiple data storage technologies

Selected based on the way data is being used by individual applications Why store binary images in relational databases, when there are

better storage systems?

Can occur both over the enterprise as well as within a single application

10.10.2012

2012 © Trivadis


Polyglot Persistence

Today we use the same database for all kind of data

• Business transactions, session management data, reporting, logging information, content information, ...

No need for same properties of availability, consistency or backup requirements

Polyglot Data Storage Usage allows to mix and match Relational and NoSQL data stores

10.10.2012

Polygot Persistence Model

E-commerce Application

Shopping cart data User Sessions Product Catalog RecomendationsCompleted Order

Key-Value RDMBS Document Graph

„Traditional“ Persistence Model


RDBMS


2012 © Trivadis


Polyglot Persistence – Challenges

• Decisions• Have to decide what data storage technology to use• Today it‘s easier to go with relational

• New Data Access APIs• Each data store has its own mechanisms for accessing the data• Different API’s

• Solution: Wrap the data access code into services (Data/Entity Service) exposed to applications

• Will enforce a contract/schemato a schemaless database

10.10.2012

Service-Oriented Polygot Persistence Model

User Session Service

Shopping CartService Order Service

Product CatalogService

Recomendation Service



Key-Value RDMBS Document Graph

2012 © Trivadis


Polyglot Persistence – Challenges

• Immaturity• NoSQL tools are still young, full of rough edges that new tools have• Not much experience, we don‘t know how to use them well• No patterns and best practices exist yet

• Organizational Change• How will the different data groups in an enterprise react to this new

technology

• Dealing with eventual consistency paradigm• Reaction of different stakeholders to the fact that data could be

stale• How to enforce rules to sync data across systems

10.10.2012

2012 © Trivadis


Agenda

1. What is NoSQL and Big Data




5. Summary

10.10.2012

2012 © Trivadis


Data Access Architecture for Polyglot Persistence

well known design patterns are still valid!

some best practices we know in data access are still valid!

10.10.2012

O/R Mapping

RDBMS

Repository/DAO

Service

Consumer

REST/SOAP

SQL

REST API

NoSQL NoSQL

Consumer

REST Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Middle TierDomainService Application Integration

Data Transfer Objects (DTO)

Web

Ser

vice

Ex

po

rter

Ap

plic

atio

n S

erv

ice

Bea

n

Domain Objects O/R Mapping

SQL API

NoSQL APIRepository Bean

Factory Bean

DAO Bean

Integration

Domain Service Bean

SOAP

REST

Aggregate

Res

ou

rce

Tie

r

Consumer

Co

mp

os

ite

Ap

plic

atio

n

Middle Tier Architecture for Polyglot Persistence

10.10.2012

2012 © Trivadis


Polyglot Persistence with Spring Data

makes it easier to build Spring-powered applications that use new data access technologies

provide improved support for relational database technologies

Commons project supports Polyglot Persistence

Currently support for:• JPA and JDBC (relational)• Apache Hadoop• GemFire• REST• Redis• MongoDB• Neo4J• Hbase

10.10.2012

NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Spring Data – Mapping to Relational Database (using JPA)

10.10.2012

Annotations define the mapping:@Entity, @Id, @Column,@OneToOne, @OneToMany, @JoinColumn,

NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Spring Data – Mapping to Relational Database

10.10.2012

<jpa:repositories base-package="com.oreilly.springdata.jpa" />

<bean class="org.springframework.orm.jpa.JpaTransactionManager"> <property name="entityManagerFactory" ref="entityManagerFactory" /></bean>

<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"> <property name="dataSource" ref="dataSource" /> <property name="packagesToScan" value="com.oreilly.springdata.jpa" /></bean>

@Repository@Profile(“jpa")class JpaCustomerRepository implements CustomerRepository { @Override public Customer findByEmailAddress(EmailAddress emailAddress) {

TypedQuery<Customer> query = em.createQuery( "select c from Customer c where c.emailAddress = :email“,

Customer.class); query.setParameter("email", emailAddress);

return query.getSingleResult(); }

public interface CustomerRepository extends Repository<Customer, Long> { Customer findByEmailAddress(EmailAddress emailAddress);}

Customer guido= repository.findByEmailAddress(new EmailAddress(“[email protected]"));

Customer anotherCust= new Customer(“Peter", “Sample");anotherCust.setEmailAddress(guido.getEmailAddress());

repository.save(anotherCust);

NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Spring Data – Mapping to MongoDB

10.10.2012

Annotations define the mapping:@Document, @Id, @Indexed,@PersistenceConstructor, @CompoundIndex, @DBRef, @GeoSpatialIndex, @Value

NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Spring Data – Generic Repositories for MongoDB

10.10.2012

<mongo:db-factory id="mongoDbFactory" dbname="e-store" />

<mongo:mapping-converter id="mongoConverter" base-package="com.oreilly.springdata.mongodb"><mongo:custom-converters base-package="com.oreilly.springdata.mongodb" /></mongo:mapping-converter>

<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"> <constructor-arg ref="mongoDbFactory" /> <constructor-arg ref="mongoConverter" /> <property name="writeConcern" value="SAFE" /></bean>

<mongo:repositories base-package="com.oreilly.springdata.mongodb" />

@Repository@Profile("mongodb")class MongoDbCustomerRepository implements CustomerRepository { @Override public Customer findByEmailAddress(EmailAddress emailAddress) {

Query query = query(where("emailAddress").is(emailAddress));return operations.findOne(query, Customer.class);

}

public interface CustomerRepository extends Repository<Customer, Long> { Customer findByEmailAddress(EmailAddress emailAddress);}




NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Spring Data – Mapping to Neo4J

10.10.2012

Annotations define the mapping:@NodeEntity, RelationShipEntity, @GraphId, @RelatedTo, @RelatedToVia, @EndNode, @Fetch,

Customer

Address

Country

ADDRESS

COUNTRY

Order

BILLING_ADDRESS

DELIVERY_ADDRESS

Product

Tag

RATED TAG

LINE_ITEM

2012 © Trivadis


Spring Data – Generic Repositories for MongoDB

10.10.2012

<neo4j:config graphDatabaseService="graphDatabaseService" /><neo4j:repositories base-package="com.oreilly.springdata.neo4j" />

<bean id="graphDatabaseService" class="org.neo4j.kernel.EmbeddedGraphDatabase" destroy-method="shutdown"> <constructor-arg value="target/graph.db" /></bean>

public interface CustomerRepository extends GraphRepository<Customer> { Customer findByEmailAddress(EmailAddress emailAddress);}




NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis


Expose contract-first Web service

Use any Java Web Service Framework which supports Contract-First approach

Can be SOAP or can be REST

Maps the data contract (WSDL or WADL) to the schemaless database

Uses the different Repository implementations

Must handle data migration issues together with the Repository

10.10.2012

NoSQL

Repository/DAO

Service

Consumer

REST/SOAP

???

2012 © Trivadis41

NoSQL databases asCaching in a SOA

Results are returned from cache rather than invoking always the external service Product data is rather static, so ideal candidate for caching

Product DB

Oracle ServiceBus

Proxy Service

Business Service

1

2 3

ResultCache

10.10.2012NoSQL Databases for Implementing Data Services – Should I Care?

2012 © Trivadis


Schemaless – We still have to migrate the data!

With RDMBS we are used to keep DDL scripts together with DML scripts for each single data model change

• Has to be in sync with the data access code

RDBMS has to be changed before the applicationis changed => possible application downtime

• This is what the schemaless approach of most NoSQL DB tries to avoid

Schemaless DBs still need careful migration, due to implicit schema in any data access code

But a more “on-demand” approach is possible• Code can read data in a way that it tolerant to

changes in the data’s implicit schema and migratethe data on the next update

• Similar to service versioning => gradual change

10.10.2012

Customer

First Name: PeterLast Name: SampleBilling Address


Customer

Name: Peter Sample

BillingAddress

Street: Somestreet 10City: SomewherePostalCode: 55901

Customer


Billing Address


Customer

Name: Peter SampleFirstName: PeterLastName: Sample

Billing Address


Version 1.0

Version 2.0

Transition Version 1.0 => 2.0

2012 © Trivadis


Pros & Cons of NoSQL compared to RDBMS

10.10.2012

Cons

• Lacks in tool and framework support

• Few other implementations => potential lock in

• No support for ad-hoc queries

• Another/A new database in production to take care of

Pros

• No O/R impedance mismatch

• Can easily evolve schemas

• Can represent semi-structured info

• Can represent graphs/networks (with performance)

2012 © Trivadis


Summary

Relational databases are here to stay but NoSQL offers new persistence model

Polyglot Persistence will be the future

Schemaless does not mean there is no data migration! => but a more on-demand model might be possible

Encapsulate data access code to be able to switch databases

Service-orientation provides the data contract to a NoSQL database => to make information reusable

Don‘t commit to a NoSQL until you have done a significant PoC

Make sure that Operations people (DBAs) are on board early enough

Non-relational is not new in an enterprise (OLTP vs. OLAP)

10.10.2012

2012 © Trivadis


BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

VIELEN DANK.Trivadis

Guido [email protected]

[email protected]

10.10.2012

Five Cool use cases for the Spring component of the Oracle SOA SuiteMonday 3th December at 11:50 - 12:35 – Executive 7

Where and When Should I Use the Oracle Service BusTuesday 4th December at 09:55 - 10:55 - Hall 8b

NoSQL Databases for Implementing Data Services – Should I Care?

Documents

Transcript of NoSQL Databases for Implementing Data Services – Should I Care?