Introduction   Couchbase Server 2.0

download Introduction    Couchbase Server 2.0

of 38

  • date post

    07-Feb-2016
  • Category

    Documents

  • view

    37
  • download

    0

Embed Size (px)

description

Introduction à Couchbase Server 2.0. Tugdual Grall TechnicalEvangelist @ tgrall. {“about” : “me”}. Tugdual “Tug” Grall Couchbase Technical Evangelist eXo CTO Oracle Developer/Product Manager Mainly Java/ SOA Developer in consulting firms. Web @ tgrall http://blog.grallandco.com - PowerPoint PPT Presentation

Transcript of Introduction   Couchbase Server 2.0

Introduction Couchbase Server

1Introduction Couchbase Server 2.0Tugdual GrallTechnicalEvangelist@tgrall{about : me}Tugdual Tug GrallCouchbaseTechnical EvangelisteXoCTOOracleDeveloper/Product ManagerMainly Java/SOADeveloper in consulting firmsWeb@tgrallhttp://blog.grallandco.comTgrall

NantesJUG co-founder

Pet Project :http://www.resultri.com

AgendaIntroduction to Couchbase ServerCluster ManagementOperation and Document DesignIntroduction to Views and QueriesCross Data Center ReplicationQ&ACouchbase Developer Day Paris is this Friday! http://couchbase.com/eventsCouchbase Server 2.0Couchbase Server

NoSQL Document DatabaseCouchbase Open Source ProjectLeading NoSQL database project focused on distributed database technology and surrounding ecosystemSupports both key-value and document-oriented use casesAll components are available under the Apache 2.0 Public LicenseObtained as packaged software in both enterprise and community editions.Couchbase Open Source Project

Easy ScalabilityConsistent High PerformanceAlways On 24x365Grow cluster without application changes, without downtime with a single clickConsistent sub-millisecond read and write response times with consistent high throughputNo downtime for software upgrades, hardware maintenance, etc.

Flexible Data ModelJSON document model with no fixed schema.Couchbase ServerNew in 2.0JSON supportIndexing and QueryingCross data center replicationIncremental Map Reduce

JSON support natively stored as json, whne you build an app, there is not conversion required. New doc viewing , editing capability. Indexing and querying look inside your json, build views and query for a key, for ranges or to aggregate data Incremental mapreduce powers indexing. Build complex views over your data. Great for real-time analytics XDCR replicate information from one cluster to another cluster 9

HeartbeatProcess monitorGlobal singleton supervisorConfiguration manageron each nodeRebalance orchestratorNode health monitorone per clustervBucket state and replication managerhttpREST management API/Web UIHTTP8091Erlang port mapper4369Distributed Erlang21100 - 21199Erlang/OTPstorage interfaceCouchbase EP Engine11210Memcapable 2.0Moxi11211Memcapable 1.0MemcachedNew Persistence Layer8092Query APIQuery EngineData ManagerCluster ManagerCouchbase Server 2.0 Architecture

New Persistence Layerstorage interfaceCouchbase EP Engine11210Memcapable 2.0Moxi11211Memcapable 1.0Object-level CacheDisk Persistence8092Query APIQuery EngineHTTP8091Erlang port mapper4369Distributed Erlang21100 - 21199HeartbeatProcess monitorGlobal singleton supervisorConfiguration manageron each nodeRebalance orchestratorNode health monitorone per clustervBucket state and replication managerhttpREST management API/Web UIErlang/OTPServer/Cluster Management & Communication(Erlang)RAM Cache, Indexing & Persistence Management(C & V8)The Unreasonable Effectiveness of C by Damien KatzCouchbase Server 2.0 ArchitectureThe network/cluster management is written in Erlang with several components:- OTP : Open Telecom Platform- all the administartion console is created in Erlang : - REST API - Web UI (many JS with JQuery and Sparkline)

- The data server is written in CIncludes memcache as main cache for the DBThe storage itself is also written in C, based on a Append only

332Single node - Couchbase Write Operation

Managed CacheDisk QueueDiskReplication QueueApp ServerCouchbase Server NodeDoc 1Doc 1Doc 1To other node1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk12GETDoc 1332Single node - Couchbase Read Operation

Disk QueueReplication QueueApp ServerDoc 1Doc 1Doc 1Managed CacheDiskTo other nodeCouchbase Server Node1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk13Couchbase Server ClusterBasic OperationDocs distributed evenly across servers Each server stores both active and replica docsOnly one server active at a timeClient library provides app with simple interface to databaseCluster map provides map to which server doc is onApp never needs to knowApp reads, writes, updates docsMultiple app servers can access same document at same timeUser Configured Replica Count = 1READ/WRITE/UPDATEActiveDoc 5Doc 2DocDocDocServer 1ActiveDoc 4Doc 7DocDocDocServer 2Doc 8ActiveDoc 1Doc 2DocDocDocREPLICADoc 4Doc 1Doc 8DocDocDocREPLICADoc 6Doc 3Doc 2DocDocDocREPLICADoc 7Doc 9Doc 5DocDocDocServer 3Doc 6App Server 1COUCHBASE Client Library

Cluster MapCOUCHBASE Client Library

Cluster MapApp Server 2Doc 9Bulletize the text. Make sure the builds work.14Add Nodes to ClusterTwo servers addedOne-click operationDocs automatically rebalanced across clusterEven distribution of docsMinimum doc movementCluster map updatedApp database calls now distributed over larger number of servers

REPLICAActiveDoc 5Doc 2DocDocDoc 4Doc 1DocDocServer 1REPLICAActiveDoc 4Doc 7DocDocDoc 6Doc 3DocDocServer 2REPLICAActiveDoc 1Doc 2DocDocDoc 7Doc 9DocDocServer 3Server 4Server 5REPLICAActiveREPLICAActiveDocDoc 8DocDoc 9DocDoc 2DocDoc 8DocDoc 5DocDoc 6READ/WRITE/UPDATEREAD/WRITE/UPDATEApp Server 1COUCHBASE Client Library

Cluster MapCOUCHBASE Client Library

Cluster MapApp Server 2Couchbase Server ClusterUser Configured Replica Count = 1Bulletize the text. Make sure build work properly.15Fail Over NodeREPLICAActiveDoc 5Doc 2DocDocDoc 4Doc 1DocDocServer 1REPLICAActiveDoc 4Doc 7DocDocDoc 6Doc 3DocDocServer 2REPLICAActiveDoc 1Doc 2DocDocDoc 7Doc 9DocDocServer 3Server 4Server 5REPLICAActiveREPLICAActiveDoc 9Doc 8DocDoc 6DocDocDoc 5DocDoc 2Doc 8DocDocApp servers accessing docsRequests to Server 3 failCluster detects server failedPromotes replicas of docs to activeUpdates cluster mapRequests for docs now go to appropriate serverTypically rebalance would followDocDoc 1Doc 3App Server 1COUCHBASE Client Library

Cluster MapCOUCHBASE Client Library

Cluster MapApp Server 2User Configured Replica Count = 1Couchbase Server ClusterBulletize the text. Make sure build work properly.

16Demo Time17Operation & Document Design18Basic Store & Retrieve Operationsget(key)Retrieve a document

set(key, value)Store a document or replace if it existsadd(key, value)Store a document and fail if it existsreplace(key, value)Replace a document and fail if it does not exist

Lots of other OperationsAtomic Counters (increment, decrement)Append/PrependCAS (Compare and Set)Optimistic LockingGet with LockWrite Lock on ObjectsBulk OperationsSaves network overheadStatsView operations

Couchbase 2.0 Bonus Points: JSONJSON is a lightweight format to represent document structure in a language-independent manner.

If JSON documents are stored, the View engine can be used.

Allows to build secondary indexes on your datasets.

Makes it possible to ask questions likeGive me all user documents by lastname.Flexible Data ModelNo need to worry about the database when changing your applicationRecords can have different structures, there is no fixed schemaAllows painless data model changes for rapid application development

{ ID: 1, FIRST: Dipti, LAST: Borkar, ZIP: 94040, CITY: MV, STATE: CA}JSONJSONJSONJSONRapid app dev withouth the need to perform an expensive alter table operation. 22Aggregate View of Data

http://martinfowler.com/bliki/AggregateOrientedDatabase.htmlStore and Retrieve AggregatesEasier to Distribute DataMore FlexibilityReduced Latencyorder::1001{uid: ji22jd,customer: Ann,line_items: [ { sku: 0321293533, quan: 3, unit_price: 48.0 },{ sku: 0321601912, quan: 1, unit_price: 39.0 },{ sku: 0131495054, quan: 1, unit_price: 51.0 } ],payment: { type: Amex, expiry: 04/2001, last5: 12345 }}

Demo Time25Official SDKsCommunity SDKswww.couchbase.com/develop

Python

Ruby

Go

Clojure

Indexing and Querying27Indexing and Querying The basicsDefine materialized views on JSON documents and then query across the data set Using views you can definePrimary indexes Simple secondary indexes (most common use case)Complex secondary, tertiary and composite indexesAggregations (reduction) Indexes are eventually indexed Queries are eventually consistentBuilt using Map/Reduce technology Map and Reduce functions are written in Javascript

Couchbase Server ClusterIndexing and Querying User Configured Replica Count = 1ActiveDoc 5Doc 2DocDocDocServer 1REPLICADoc 4Doc 1Doc 8DocDocDocApp Server 1COUCHBASE Client Library

Cluster MapCOUCHBASE Client Library

Cluster MapApp Server 2Doc 9Indexing work is distributed amongst nodesLarge data set possibleParallelize the effortEach node has index for data stored on itQueries combine the results from required nodesActiveDoc 5Doc 2DocDocDocServer 2REPLICADoc 4Doc 1Doc 8DocDocDocDoc 9ActiveDoc 5Doc 2DocDocDocServer 3REPLICADoc 4Doc 1Doc 8DocDocDocDoc 9QueryBulletize the text. Make sure the builds work.29Demo Time30Cross Data Center Replication(XDCR)31Cross Data Center Replication The basicsReplicate your Couchbase data across clustersClusters may be spread across geosConfigured on a per-bucket (per-database) basisSupports unidirectional and bidirectional operationApplication can read and write from both clusters Active Active replicationReplication throughput scales out linearlyDifferent from intra-cluster replicationOverview of what this feature is32Cross Data Center Replication (XDCR)Couchbase Server ClusterUSActiveDoc Doc 2Server 1Doc 9Serv