Post on 20-Aug-2015
1
A Technical Overview of Couchbase using Java
Raghavan “Rags” Srinivas Developer Advocate
Simple. Fast. Elas.c.
2
• Developer Advocate at Couchbase working with developers
• Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences
• Taught undergrad and grad courses • Technology Evangelist for Java and Sun Microsystems for 10+ years
• SKll trying to understand and work more effecKvely on Java and distributed systems
• Philosophy: “Be#er to have an unanswered ques1on than a unques1oned answer”
Speaker CredenKals
3
• Distributed Systems and Performance • Couchbase Server • GePng Started • DemonstraKon and Case Study • Resources and Summary
Agenda
5
Web Application Architecture and Performance
Application Scales Out Just add more commodity web servers
Database Scales Up Get a bigger, more complex server
www.facebook.com/animalparty
Web Servers
RelaKonal Database
Load Balancer
-‐ Expensive and disrupKve sharding -‐ Doesn’t perform at Web Scale
7
• The Network is reliable • Latency is zero • Bandwidth is infinite • The Network is secure • Topology doesn’t change • There is one administrator • Transport cost is zero • The Network is homogenous
Eight Fallacies of Distributed CompuKng
8
Couchbase data layer scales like application logic tier Data layer now scales with linear cost and constant performance.
Application Scales Out Just add more commodity web servers
Database Scales Out Just add more commodity data servers
Scaling out flattens the cost and performance curves.
Couchbase Servers
www.facebook.com/animalparty
Web Servers Load Balancer
Horizontally scalable, schema-‐less, auto-‐sharding, high-‐performance at Web Scale
10
Couchbase “write” Data Flow – applicaKon view
User acKon results in the need to change the VALUE of KEY
ApplicaKon updates key’s VALUE, performs SET operaKon
Couchbase client hashes KEY, idenKfies KEY’s master server SET request sent over
network to master server
Couchbase replicates KEY-‐VALUE pair, caches it in memory and stores it to disk
1
2
3 4
5
11
Couchbase Data Flow – under the hood
SET request arrives at KEY’s master server
Listener-‐Sender
Master server for KEY Replica Server 2 for KEY Replica Server 1 for KEY
2 2
1 SET acknowledgement returned to applicaKon 3
Disk Disk Disk
RAM*
Couchb
ase storage en
gine
Disk Disk Disk
2
4
12
Proven at Small, and Extra Large Scale
• Leading cloud service (PAAS) provider
• Over 150,000 hosted applications • Couchbase Server serving over
6,200 Heroku customers
• Social game leader – FarmVille, Mafia Wars, Empires and Allies, Café World, FishVille
• Over 230 million monthly users • Couchbase Server is the
primary database behind key Zynga properties
14
Java Client SDK
.Net SDK
PHP SDK
Ruby SDK
…
spymemcached ConnecKon
HTTP couchDB connecKon
Java client API
User Code
Couchbase Server
List <URI> servers = new ArrayList<URI>(); servers.add(oneserver); servers.add(twoserver); CouchbaseClient cb = new CouchbaseClient(servers, "aBucket", "letmein"); // this is all the same as before cb.set("hello", 0, "world"); cb.get("hello"); Map<String, Object> manyThings = cb.getBulk(CollecKon<String> keys); // accessing a view View view = cb.getView("design_document", "my_view"); Query query = new Query(); query.getRange("abegin", "theend");
hvp://www.couchbase.com/develop
Couchbase Client Java SDK
15
Client at RunKme: Adding a node
15
Couchbase Client
Couchbase Server Node
Couchbase Server Node
Couchbase Server Node
Couchbase Server Node
Couchbase Server Node
Cluster ���Topology���Update
New node coming online
16
Asynchronous OperaKons
• Set OperaKon // Do an asynchronous set! OperationFuture<Boolean> setOp = client.set(KEY, EXP_TIME, VALUE);!! // Block and check to see if our set succeeded! try {! if (setOp.get().booleanValue()) {! System.out.println("Set Succeeded");! } else {! System.err.println("Set failed: " + ! setOp.getStatus().getMessage());! }! } catch (Exception e) {! System.err.println("Exception while doing set: " +! e.getMessage());! }!
17
OperaKons Available to a Client of Couchbase Server
• Store OperaKons – Add: Store the document if it does not yet exist – Set: Store the document, overwriKng exisKng if necessary
• Retrieve OperaKons – Get: Fetch the document – Get and touch: Fetch the document and update the TTL – MulKget: Fetch mulKple documents at the same Kme
• Update operaKons – Append/prepend: Add data in front of or on the end of a document – Delete: Remove the document from the store – Compare and Swap (CAS): Replace the current document, if CAS matches – Replace: Replace the document if it exists, otherwise do not – Touch: Update the TTL for a document
18
Document Metadata: • TTL • CAS value • Flags
Q: What happens to a document persisted a<er it’s TTL?
A: A regular background job will remove expired documents.
Q: How do I use flags?
A: Frequently used to idenKfy data type or other avributes, such as compression. Behavior varies from client to client.
Metadata
19
Distributed System Design: Concurrency Controls
• Compare and Swap OperaKons – Owen referred to as “CAS” – OpKmisKc concurrency control – Available with many mutaKon
operaKons, depending on client.
• Get with Lock – Owen referred to as “GETL” – PessimisKc concurrency control – Locks have a short TTL – Locks released with CAS
operaKons – Useful when working with object
graphs
19
Actor 1 Actor 2
Couchbase Server
CAS mismatch Success
A
B
F
C D
E
21
3 Objects (documents) within game: • Players • Monsters • Items
Gameplay:
• Players fight monsters • Monsters drop items • Players own items
Demo: The next big social game
22
{ "_id": "Keith4540", "_rev": "1-‐ab354009ce09f198c555b693e057adce", "jsonType": "player", "uuid": "35767d02-‐a958-‐4b83-‐8179-‐616816692de1", "name": "Keith4540", "hitpoints": 75, "experience": 663, "level": 4, "loggedIn": false
}
Player Document
23
{ "_id": "Katana_e5890c94-‐11c6-‐48-‐65746ce6c560", "_rev": "1-‐d6bbd5e814c32c66e22db2918a2efcd9", "jsonType": "item", "name": "Katana_e5890c94-‐11c6-‐65746ce6c560", "uuid": "e5890c94-‐11c6-‐4856-‐a7a6-‐65746ce6c560", "ownerId": "Dale9887"
}
Item Document
Player “_id”
24
Monster Document
{ "_id": "Bauchan9932", "_rev": "1-‐5c90be58be58134a0fc5e7db77dab5f2", "jsonType": "monster", "name": "Bauchan9932", "uuid": "d10dfc1b-‐0412-‐4140-‐b4ec-‐affdbf2aa5ec", "hitpoints": 370, "experienceWhenKilled": 52,
}
27
Use case -‐ simple farming game: • A player can have a variety of plants on their farm. • A player can add or remove plants from their farm. • A Player can see what plants are on another player's farm.
Tribal Crossing: RepresenKng Game Data in Couchbase
28
Tribal Crossing: Challenges
● Write-‐heavy requests – Caching does not help – MySQL / InnoDB limitaKon (Percona)
● Need to scale drasKcally over night – My Polls – 100 to 1m users over a weekend
● Small team, no dedicated sysadmin – Focus on what we do best – making games
● Keeping cost down
29
Tribal Crossing: RepresenKng Game Data in Couchbase
RepresenKng Objects ● Simply treat an object as a Map/dicKonary/JSON object ● Determine the key for an object using the class name (or type) of the object and an unique ID
RepresenKng Object Lists
● DenormalizaKon ● Save a comma separated list or an array of object IDs
30
Tribal Crossing: RepresenKng Game Data in Couchbase
Player Object Key: 'Player1' JSON { “_id” : “Player1”, “nid” : 1, “name” : “Shawn” }
Plant Object Key: 'Plant201' JSON { “_id” : “Plant201”, “nid” : 201, “player_id” : 1 “name” : “Starflower” }
PlayerPlant List Key: 'Player1_PlantList' JSON { “_id” : “Player1_Plantlist”, “plants” : [201, 202, 204] }
31
Tribal Crossing: Schema-‐less Game Data
● No need to “ALTER TABLE” ● Add new “fields” all objects at any Kme
– Specify default value for missing fields – Increased development speed
● Using JSON for data objects though, ● Offers the ability to query and analyze arbitrary fields in Couchbase 2.0
32
Tribal Crossing: Accessing Game Data in Couchbase
Get all plants that belong to a given player Request: GET /player/1/farm // Create a new PlantList from players PlantList playersPlants = new PlantList(cbclient.get("Player1_PlantList"); for (Plant plant : playersPlants) { aPlayer.addPlant(plant);
33
Tribal Crossing: Modifying Game Data in Couchbase
Give a player a new plant // Create the new plant Plant givenPlant = new Plant(100, "Mushroom"); cbclient.set("Plant100", givenPlant); // Update the player plant list Player thePlayer = player.fetch(cbclient.get("Player1"); // Add the plant to the player thePlayer.receivePlant(givenPlant); // Store the player's new plant list cbclient.set("Player1_PlantList", thePlayer.getPlantsArray());
34
• Significantly reduced the cost incurred by scaling up database servers and managing them.
• Achieved significant improvements in various performance metrics (e.g., read, write, latency, etc.)
• Allowed focus more on game development and opKmizing key metrics
• Plan to use real-‐Kme Map-‐Reduce, querying, and indexing abiliKes provided by Couchbase Server 2.0
Tribal Crossing: Conclusion
36
Couchbase Server 2.0
• Next major release of Couchbase Server • Currently in Developer Preview
What’s new: • Indexing and Querying • Incremental Map Reduce • Cross Data Center ReplicaKon
• Fully backwards compaKble with exisKng Couchbase Server
37
Accessing a View in Couchbase Server 2.0
Query query = new Query(); query.setReduce(false); query.setIncludeDocs(true); View view = cbclient.getView("dev_players", "player_by_level"); HttpFuture<ViewResponse> future = client.asyncQuery(view, query); ViewResponse response = future.get(); if (!future.getStatus().isSuccess()) { // do something to handle the errror } Iterator<ViewRow> itr = response.iterator(); while (itr.hasNext()) { ViewRow row = itr.next(); if (ITEMS.containsKey(row.getId())) { assert ITEMS.get(row.getId()).equals(row.getDocument()); } }
39
• Couchbase Server Downloads – hvp://www.couchbase.com/downloads-‐all – hvp://www.couchbase.com/couchbase-‐server/overview
• Developing with Java Client libraries – hvp://www.couchbase.com/develop/java/current
• Couchbase forums – hvp://www.couchbase.com/forums/
• Sqoop-‐based Hadoop Connector for Couchbase – hvp://www.couchbase.com/develop/connectors/hadoop
• Couch Conference (coming to a city near you) – hvp://www.couchbase.com/couchconf-‐world-‐tour
• Tribal Crossing blog – hvp://www.fablelabs.com/press/real-‐world-‐nosql-‐membase-‐at-‐tribal-‐crossing.html
Resources, Summary and Call For AcKon