Introducing the App Engine datastore

Thursday, May 26, 2011

Hands on with the App Engine Datastore

Ikai LanMay 9th, 2011

About the speaker

• Ikai Lan - Developer Programs Engineer, Developer Relations• Twitter: @ikai• Google Profile: http://profiles.google.com/ikai.lan

Lab prerequisites

• JDK 1.5+• Apache Ant• Codelab package: http://code.google.com/p/2011-datastore-

bootcamp-codelab/downloads/detail?name=2011-datastore-bootcamp-codelab.zip

Shortlink: http://tinyurl.com/datastore-bootcamp

Goals of this talk

• Understand a bit of how the datastore works underneath the hood

• Have a conceptual background for the persistence codelab

Understanding the datastore

• The underlying Bigtable• Indexing and queries• Complex queries• Entity groups• Underlying infrastructure

Datastore layers

Complex queries

Entity Group Transactions

Queries on properties

Key range scan

Get and set by key

Datastore ✓ ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓ ✓Bigtable ✓ ✓

Datastore layers

Complex queries

Get and set by key, key range scans

Datastore ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓Bigtable ✓

Complex queries

Key range scan

Get and set by key

What does a Bigtable row look like?

Source: http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/bigtable-osdi06.pdf

Bigtable API

• “Give me the column ‘name’ at key 123”• “Set the column ‘name’ at key 123 to ‘ikai’”• “Give me all columns where the key is greater than 100 and less

than 200”

Datastore layers

Complex queries

Key range scan

Get and set by key

Megastore API

• “Give me all rows where the column ‘name’ equals ‘ikai’”• “Transactionally write an update to this group of entities”• “Do a cross datacenter write of this data such that reads will be

strongly consistent” (High Replication Datastore)• Megastore paper: http://www.cidrdb.org/cidr2011/Papers/

CIDR11_Paper32.pdf

Datastore layers

Complex queries

Key range scan

Get and set by key

App Engine Datastore API

• “Give me all Users for my app where the name equals ‘ikai’, company equals ‘Google’, and sort them by the ‘awesome’ column, descending”

Queries

Let’s save an Entity with the low-level Java API DatastoreService datastore = DatastoreServiceFactory

.getDatastoreService();

Entity ikai = new Entity("User", "ikai@google.com");

ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");

ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);

Get an instance of the DatastoreServiceDatastoreService datastore = DatastoreServiceFactory

Fetch a client instance

Instantiate a new EntityDatastoreService datastore = DatastoreServiceFactory

Set the Entity Kind

Instantiate a new EntityDatastoreService datastore = DatastoreServiceFactory

Set a unique key

Set indexed propertiesDatastoreService datastore = DatastoreServiceFactory

First argument is the property name

Second argument is the property value

Set unindexed propertiesDatastoreService datastore = DatastoreServiceFactory

This property will be saved, but we will not run queries against it

Commit the entity to the datastoreDatastoreService datastore = DatastoreServiceFactory

Save the thing!

What happens when we save?

Write the entity

Write the indexes

Make the write RPC Success!

What actually gets written?

Bigtable key Value

AppId:User:ikai@google.com ( Protobuf serialized entity - includes firstName, company and biography values )

Read more: http://code.google.com/appengine/articles/storage_breakdown.html

Bigtable key Value

AppId:User:firstName:ikai:ikai@google.com ( Empty )

AppId:User:company:google:ikai@google.com ( Empty )

Entities table

Indexes table

Now let’s run a query

• If we have the key, we can fetch it right away by key• What if we don’t? We need indexes.

Let’s run a queryDatastoreService datastore = DatastoreServiceFactory

Query queryByName = new Query("User");

queryByName.addFilter("firstName", FilterOperator.EQUAL, "ikai");

List<Entity> results = datastore.prepare( queryByName).asList( FetchOptions.Builder.withDefaults());

// Roughly equivalent to: // SELECT * from User WHERE firstname = ‘ikai’;

Step 1: Query the indexes table

Bigtable key Value

Entities table

Indexes table

Scan the indexes table for values >= AppId:User:firstName:

Step 2: Start extracting keys

Bigtable key Value

Entities table

Indexes table

That gets us this row - extract the key ikai@google.com

Step 3: Batch get the entities themselves

Bigtable key Value

Entities table

Indexes tableNow let’s go back to the entities table and fetch that key. Success!

Key takeaways

• This isn’t a relational database– There are no full table scans– Indexes MUST exist for every property we want to query– Natively, we can only query on matches or startsWith queries– Don’t index what we never need to query on

• Get by key = one step. Query on property value = 2 steps

Let’s run a more complex query!DatastoreService datastore = DatastoreServiceFactory

Query queryByName = new Query("User");

queryByName.addFilter("firstName", FilterOperator.EQUAL, "ikai");

queryByName.addFilter("company", FilterOperator.EQUAL, "google");

List<Entity> results = datastore.prepare( queryByName).asList( FetchOptions.Builder.withDefaults());

// Roughly equivalent to: // SELECT * from User WHERE firstname = ‘ikai’ // AND company = ‘google’;

Query resolution strategies

• This query can be resolved using built in indexes– Zig zag merge join - we’ll cover this example

• Can be optimized using composite indexes

Zig zag across multiple indexes

Bigtable key

AppId:User:firstName:alfred:alfred@acme.com

AppId:User:firstName:ikai:ikai@acme.com

AppId:User:firstName:ikai:ikai@google.com

AppId:User:firstName:ikai:ikai@megacorp.com

AppId:User:firstName:zed:zed@megacorp.com

Bigtable key

AppId:User:company:acme:alfred@acme.com

AppId:User:company:google:david@google.com

AppId:User:company:google:ikai@google.com

AppId:User:company:google:max@google.com

AppId:User:company:megacorp:zed@megacorp.com

Begin by scanning indexes >= AppId:User:company:google

Bigtable key

There’s at least a partial match, so we “jump” to the next index

Bigtable key

AppId:User:company:megacorp:zed@megacorp.comMove to the next index. Start a scan for keys >= AppId:User:firstName:ikai:david@google.com

Bigtable key

AppId:User:company:megacorp:zed@megacorp.comOkay, so that’s a twist. The first value that matches has key ikai@google.com! Does this value exist in the first index?

Bigtable key

Let’s advance the original cursor to >= AppId:User:company:google:ikai@google.com

Bigtable key

Alright! We found a match. Let’s add the key to our in memory list and go back to the first index

Bigtable key

Let’s move on to see if there are any more matches. Let’s start at max@google.com

Bigtable key

Are there any keys >= AppId:User:firstName:ikai:max@google.com?

Bigtable key

No. We’re at the end of our index scans. Let’s do a batch key of our list of keys: [ ‘ikai@google.com’ ]

Bigtable key

Batch get the entities themselves

Bigtable key Value

Entities table

Now let’s go back to the entities table and fetch that key. Success!

Let’s change the shape of the data

• Zig zag performance is HIGHLY dependent on the shape of the data

• Let’s go ahead and muck with the data a bit

Same query, sparsely distributed matches

Bigtable key

AppId:User:firstName:igor:ikai@google.com

Bigtable key

Begin by scanning indexes >= AppId:User:company:google

Bigtable key

Move to the next index. Start a scan for keys >= AppId:User:firstName:ikai:david@google.com

Bigtable key

Oh ... no matches. Let’s move back to the first index and move the cursor down

Bigtable key

Okay, we’ve got another Googler

Bigtable key

Move to the next index. Start a scan for keys >= AppId:User:firstName:ikai:ikai@google.com

Bigtable key

Oh ... no matches here either. Let’s go back to the first index.

Bigtable key

Oh ... no matches here either. Let’s go back to the first index.

... if these indexes were huge, we could be here for a while!

What happens in this case?

• If we traverse too many indexes, the datastore throws a NeedIndexException

• We’ll want to build a composite index

Composite index

Bigtable key

AppId:User:company:acme:firstName:alfred:alfred@acme.com

AppId:User:company:google:firstName:david:david@google.com

AppId:User:company:google:firstName:ikai:ikai@google.com

AppId:User:company:google:firstName:max:max@google.com

AppId:User:company:megacorp:firstName:zed:zed@megacorp.com

Composite index

Bigtable key

Search for all keys >= AppId:User:company:google:firstName:ikai

Composite index

Bigtable key

Well, that was much faster, wasn’t it?

Composite index tradeoffs

• Created at entity save time - incurs additional datastore CPU and storage quota

• You can only create 200 composite index• You need to know the possible queries ahead of time!

Complex Queries takeaways

• This isn’t a relational database – There are no full table scans– Indexes MUST exist for every property we want to query

• Performance depends on the shape of the data• Worse case scenario: if your query matches are highly sparse• Build composite indexes when you need them

Entity Groups

Why entity groups?

• We can perform transactions within this group - but not outside• Data locality - data are stored “near” each other• Strongly consistent queries when using High Replication

datastore within this entity group

Entity groups and transactions

• A hierarchical structuring of your data into Megastore’s unit of atomicity

• Allows for transactional behavior - but only within a single entity group

• Key unit of consistency when using High Replication datastore

Example: Data for a blog hosting service

Comment

Has manyHas many

Has many

Example: Data for a blog hosting service

Comment

Has manyHas many

Has many

This can be structured as an entity group (tree structure)!

Structure this data as an entity group

Entry Entry

CommentCommentComment

Entity group root

How are entity groups stored?

Bigtable key ValueAppId:User:ikai@google.com ( Protobuf serialized User )

AppId:User:ikai@google.com/Blog:123 ( Protobuf serialized Blog )

AppId:User:ikai@google.com/Blog:123/Entry:456 ( Protobuf serialized Entry )

AppId:User:ikai@google.com/Blog:123/Entry:456/Comment:111

( Protobuf serialized Comment )

Read more: http://code.google.com/appengine/docs/python/datastore/entities.html

Entities table

Entities table Entity groups have a single root entity

Entities table

Child entities embed the entire ancestry in their keys

Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory

Entity ikai = new Entity("User", "ikai@google.com"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();

Create the root entity

This is the first child entity - notice the third argument, which specifies the parent entity key

The next deeper entity sets the blog as the parent

We can also opt to not provide a key name and just use a parent key for a new entity

Start a new transaction

Put the entities in parallel

Actually commit the changes

Step 1: Commit

Commit Changes to entities visible

Changes to entities and indexes visible

Roll the timestamp forward on the root entity

Step 2: Entity visible

On read, check for the most recent timestamp on the root entity

This is the version we want since it represents a complete write

Step 3: Indexes updated

Indexes are written - now we can query for this entity with the new properties

Entity group and transactions takeaways

• Structure data into hierarchical trees– Large enough to be useful, small enough to maximize

transactional throughput

• Transactions need an entity group root - roughly 1 transaction/second– If you write N entities that are all part of 1 entity group, it counts as

1 write

• Optimistic locking used - can be expensive with a lot of contention

General datastore tips

• Denormalize as much as possible– As much as possible, treat datastore as a key-value store

(Dictionary or Map like structure)– Move large reporting to offline processing. This lets you avoid

unnecessary indexes

• Use entity groups for your data• Build composite indexes where you need them - “need” depends

on shape of your data

Questions?

Introducing the App Engine datastore

Technology

Transcript of Introducing the App Engine datastore

Introducing Localytics - App Analytics and Marketing Platform

Introducing App Engine for Business

Introducing the Haha App! Presentation for Product Hunt Sydney.

INTRODUCING THE NHS COVID-19 APP · 3 NHS COVID-19 App Benefits of app contact tracing The importance of the app App data The app helps trace app users who have spent time near other

Introducing the new SharePoint 2013 app model

Introducing Voltga: social app for gay guys

Introducing INFINITE CAB Taxi Dispatch software & Mobile App

Smartphone App INTRODUCING THE WELLWORKS FOR YOU SMARTPHONE APP FOR iPHONE AND ANDROID USERS.

Introducing the Common App Online For First-Year Admission Last updated: 8.9.10.

Datastore Edition Mixin' it Up - Oestrich · 2020-02-17 · Datastore Edition Chris Cahoon Eric Oestrich. Using only an object store on a largeish app and then remembering you can

Introducing WSO2 App Factory 2.0

Introducing the WSO2 App Factory

Introducing Watch Connectivity - Apple Developer...News App Social Networking App iOS Game Image Editing App Run Tracker App Weather App Surﬁng App Sports App WatchConnectivity NSURLSession

Introducing the new Maryland WIC Mobile App WIC Vendor … Flyer for Vendors_FINAL.pdfMaryland WIC Mobile App Introducing the new... WIC Vendor APL Checker The APL Checker tells you

Google App Enginepusztai/szakdolgozatok/lzs_2012_b.pdf · •App Engine Datastore •Google Cloud SQL •Google Cloud Storage •DataNucleus Access Platform •Datastore API •JDO

Introducing LucidWorks App for Splunk Enterprise webinar

Data Migrations in the App Engine Datastore

Introducing the CRS Audio Visual Design app

INTRODUCING THE IMCO APP 10TH ANNIVERSARY OF THE DIRT ...

Introducing publishing management app Bibliocloud's redesign, Spring 2016