Building RightScale's Globally Distributed Datastore - RightScale Compute 2013
Introduction to Datastore
-
Upload
software-park-thailand -
Category
Technology
-
view
860 -
download
2
description
Transcript of Introduction to Datastore
![Page 1: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/1.jpg)
Introducion to Datastore
Assoc.Prof. Dr.Thanachart NumnondaAsst.Prof. Thanisa Kruawaisayawan
Mini Master of Java TechnologyKMITL
July 2010
![Page 2: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/2.jpg)
Agenda
What is DataStore?Using DataStoreJPA in DataStore
![Page 3: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/3.jpg)
What is DataStore?
![Page 4: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/4.jpg)
What is Datastore?Google App Engine Datastore is a schema-less persistence
system, whose fundamental persistence unit is called Entity, composed by an immutable Key and a collection of mutable properties.
Entities can be created, updated, deleted, loaded by key and queried for properties values.
DataStore is consistent and transactional, with support to current transaction.
![Page 5: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/5.jpg)
The DataStoreThe Datastore is not a relational database nor a
façade.Relational database technology doesn’t scale
horizontally– Connection pools, shared caching are a problem
The Datastore is one of many public APIs used for accessing Google’s
![Page 6: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/6.jpg)
The DataStore
![Page 7: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/7.jpg)
The DataStore
![Page 8: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/8.jpg)
The DataStore : OperationsTransactions and Index are based on MegaTable.File persistence it's done with Google File System
(GFS).It's distributed by Chubby, a lock service for loosely-
coupled distributed systems.
![Page 9: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/9.jpg)
BigTableBigTable is a compressed, high performance, and
proprietary database system built on Google File System (GFS), Chubby Lock Service, and a few other Google programs
Currently not distributed or used outside of Google.BigTable development began in 2004. and is now used
by a number of Google application Google Earth, Google Map, Gmail, Youtube, etc..
![Page 10: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/10.jpg)
BigTable : DesignBigTable is a fast and extremely large-scale DBMS. It is a sparse, distributed multi-dimensional sorted map,
sharing characteristics of both row-oriented and column-oriented databases.sparse because only "not null" values are persisteddistributed in Google cloud persistent on Google File Systemmultidimensional in columns valuesordered lexicographically by key
![Page 11: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/11.jpg)
BigTable : Design
Tables are optimized for GFS by being split into multiple tablets - segments of the table.
BigTable is designed to scale into the petabyte.Each table has multiple dimensions (one of which is a
feld for time, allowing for versioning and garbage collection).
It allows an infnite number of rows and columns.
![Page 12: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/12.jpg)
Google File SystemGFS is a proprietary distributed fle system developed
by Google.It is designed to provide effcient, reliable access to
data using large clusters of commodity hardware.GFS grew out of an earlier Google effort, BigFiles,
developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford.
![Page 13: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/13.jpg)
Using DataStore
![Page 14: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/14.jpg)
DataStore OperationsDatastore operations are defned around entities (data
models) which are objects with one or more propertiesTypes: string, user, Boolean, and so onEntities may be recursive or self-referential
Entity relationships are one-to-many or many-to-many.Entities may be fxed or grow as needed.
![Page 15: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/15.jpg)
DataStore Storage ModelEvery entity is of a particular kindEntities in a kind need not have the same propertiesOne entity may have different “columns” from another in
the same kind!Unique IDs are automatically assigned unless the user
defnes a key_name
![Page 16: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/16.jpg)
Compare DataStore with Others
![Page 17: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/17.jpg)
DataStore Storage ModelBasic unit of storage is an Entity consisting of
Kind (table)Key (primary key)Entity Group (partition)0..N typed Properties (columns)
![Page 18: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/18.jpg)
Datastore QuotasEach call to Datastore counts towards the quotaThe amount of data cannot exceed the billable
Includes properties and keys but not the indices
CPU and Datastore CPU time quotas apply
![Page 19: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/19.jpg)
Using the DatastoreApplications may access the Datastore using the JDO
or the JPA classes.The JDO and JPA classes are abstracted using the
DataNucleus APIOpen source Not very popular Support for Java standards Poor documentation
![Page 20: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/20.jpg)
JPA in DataStore
![Page 21: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/21.jpg)
Setting Up JPAThe JPA and datastore JARs must be in the app's
war/WEB-INF/lib/ directory. A confguration fle named persistence.xml must be in
the app's war/WEB-INF/classes/META-INF/ directory,A confguration fle tells JPA to use the App Engine
datastore.The appengine-api.jar must also be in the war/WEB-
INF/lib/ directory.
![Page 22: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/22.jpg)
persistence.xml: Example
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">
<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>
<non-jta-data-source/>
<properties>
<property name="datanucleus.ConnectionURL" value="appengine"/>
<property name="datanucleus.NontransactionalRead" value="true"/>
<property name="datanucleus.NontransactionalWrite" value="true"/>
</properties>
</persistence-unit>
</persistence>
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">
<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>
<non-jta-data-source/>
<properties>
<property name="datanucleus.ConnectionURL" value="appengine"/>
<property name="datanucleus.NontransactionalRead" value="true"/>
<property name="datanucleus.NontransactionalWrite" value="true"/>
</properties>
</persistence-unit>
</persistence>
![Page 23: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/23.jpg)
Getting an EntityManager InstanceAn app interacts with JPA using an instance of the EntityManager.
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
public class EMF {
private static final EntityManagerFactory emfInstance =
Persistence.createEntityManagerFactory("transactions-optional");
public static EntityManagerFactory get() {
return emfInstance;
}
}
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
public class EMF {
private static final EntityManagerFactory emfInstance =
Persistence.createEntityManagerFactory("transactions-optional");
public static EntityManagerFactory get() {
return emfInstance;
}
}
![Page 24: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/24.jpg)
Entity Class : Example@Entity
public class GuestList implements Serializable {
…
@Id
private String id;
@Basic
private User author;
private String content;
@Temporal(javax.persistence.TemporalType.DATE)
private Date visitDate;
…
// Getter and Setter methods
}
@Entity
public class GuestList implements Serializable {
…
@Id
private String id;
@Basic
private User author;
private String content;
@Temporal(javax.persistence.TemporalType.DATE)
private Date visitDate;
…
// Getter and Setter methods
}
![Page 25: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/25.jpg)
Queries and IndicesA query operates on every entity of a given kind.
Specify zero or more sort ordersSpecify zero or more flters on property values
Indices are defned in the App Engine confguration flesResults are fetched directly from these indices; no indices are
created on the flyWEB-INF/datastore-indexes.xml - non-standard fles
Normalization is not recommendedOptimization techniques for RDBMSs may result in poor
Datastore performance!
![Page 26: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/26.jpg)
Query : Example EntityManager em = EMF.get().createEntityManager();
try {
Query query = em.createQuery("SELECT o FROM GuestList AS o");
@SuppressWarnings("unchecked")
List<GuestList> results = (List<GuestList>) query.getResultList();
for (Object obj : results) {
GuestList guest = (GuestList) obj;
String nickname = guest.getAuthor().getNickname();
out.println(nickname + " " + guest.getId());
}
} catch(Exception ex) {
out.println(ex);
}
EntityManager em = EMF.get().createEntityManager();
try {
Query query = em.createQuery("SELECT o FROM GuestList AS o");
@SuppressWarnings("unchecked")
List<GuestList> results = (List<GuestList>) query.getResultList();
for (Object obj : results) {
GuestList guest = (GuestList) obj;
String nickname = guest.getAuthor().getNickname();
out.println(nickname + " " + guest.getId());
}
} catch(Exception ex) {
out.println(ex);
}
![Page 27: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/27.jpg)
Entity Relationships
Models association between entities.There are four types of relationship multiplicities:
@OneToOne@OneToMany@ManyToOne
Supports unidirectional as well as bidirectional relationshipsUnidirectional relationship: Entity A references B, but B doesn't
reference A.
![Page 28: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/28.jpg)
Example : ManyToOne Mapping
![Page 29: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/29.jpg)
Example : OneToMany Mapping
![Page 30: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/30.jpg)
Transactions and Entity GroupsTransaction = Group of Datastore operations that either
succeed or failEntity groups are required because all grouped entities are
stored in the same Datastore nodeAn entity may be either created or modifed once per
transactionTransactions may fail if a different user or process tries an
update in the same group at the same timeUsers decide whether to retry or roll the transaction back
![Page 31: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/31.jpg)
Transaction in JPA : Example
Book book = em.find(Book.class, "9780596156732");
BookReview bookReview = new BookReview();
bookReview.rating = 5;
book.getBookReviews().add(bookReview);
Transaction txn = em.getTransaction();
txn.begin();
try {
book = em.merge(book);
txn.commit();
} finally {
if (txn.isActive()) {
txn.rollback();
}
}
Book book = em.find(Book.class, "9780596156732");
BookReview bookReview = new BookReview();
bookReview.rating = 5;
book.getBookReviews().add(bookReview);
Transaction txn = em.getTransaction();
txn.begin();
try {
book = em.merge(book);
txn.commit();
} finally {
if (txn.isActive()) {
txn.rollback();
}
}
![Page 32: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/32.jpg)
Unsupported Features of JPAOwned many-to-many relationships, and unowned
relationships."Join" queries.Aggregation queries (group by, having, sum, avg, max, min) Polymorphic queries.
![Page 33: Introduction to Datastore](https://reader033.fdocuments.net/reader033/viewer/2022052321/554c2177b4c905f1518b56cc/html5/thumbnails/33.jpg)
ResourcesGoogle App Engine for Java HOWTO, Andrew Lombardi, Mar
2010The Softer Side Of Schemas, Max Ross, May 2009Official Google App Engine Tutorial,
http://code.google.com/appengine/docs/java/gettingstarted/Programming Google App Engine, Don Sanderson, O'Reilly,
2010