How is OpenID helping Google? Steven Bazyl Developer Advocate .
Google App Engine Developer - Day3
-
Upload
simon-su -
Category
Technology
-
view
617 -
download
0
description
Transcript of Google App Engine Developer - Day3
GAE Developer - Day3Simon @ MiCloud2014Q1
Last Class
● Datastore basic● Datastore operation● Restrictions● Transaction
Create an Entity with KeyEntity job = new Entity("User", "[email protected]");
job.setProperty("name", "Simon Su");
job.setProperty("start", "20140103");
job.setProperty("create", new Date());
try {
Key k = ds.put(job);
out.println("Done..." + k.getId());
} catch (Exception e) {
e.printStackTrace();
}
Create an Entity with AncestorKey userKey = KeyFactory.createKey("User", "[email protected]");
Entity job = new Entity("Jobs", userKey);
job.setProperty("name", "engineer");
job.setProperty("start", new Date());
try {
Key k = ds.put(job);
out.println("Done..." + k.getId());
} catch (Exception e) {
e.printStackTrace();
}
Get data with KeyDatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Key key = KeyFactory.createKey("User", "[email protected]");
try {
Entity ent = ds.get(key);
out.print(ent);
Map m = ent.getProperties();
out.println(m.toString());
} catch (EntityNotFoundException e) {
e.printStackTrace();
}
Get data with AncestorQuery q = new Query("Jobs");
Key ancestor = KeyFactory.createKey("User", "[email protected]");
q.setAncestor(ancestor);
PreparedQuery results = ds.prepare(q);
Iterator iter = results.asIterable().iterator();
while(iter.hasNext()) {
Entity ent = (Entity) iter.next();
out.println(ent);
}
Today
● Memcache basic● Task Queue concept● Push Queue● Pull Queue
Memcache
Why memcache
● Improve Application Performance● Reduce Application Cost● Caching for Read heavy operations● Caching In Front of Datastore● Semi-durable Shared state Across App
Instances
Memcache - operationMemcacheService cache = MemcacheServiceFactory.getMemcacheService();
cache.setErrorHandler(
ErrorHandlers.getConsistentLogAndContinue(Level.INFO));
// read from cache
value = (byte[]) cache.get(key);
// save to cache
if (value == null) {
// ........
cache.put(key, value);
}
Async memcache - get// Using the asynchronous cache
AsyncMemcacheService asyncCache =
MemcacheServiceFactory.getAsyncMemcacheService();
asyncCache.setErrorHandler(
ErrorHandlers.getConsistentLogAndContinue(Level.INFO));
// read from cache
Future<Object> futureValue = asyncCache.get(key);
// ... do other work in parallel to cache retrieval
value = (byte[]) futureValue.get();
Async memcache - putif (value == null) {
// get value from other source
// ........
// asynchronously populate the cache
// Returns a Future<Void> which can be used to block until completion
asyncCache.put(key, value);
}
Using jcache - initimport java.util.Collections;
import net.sf.jsr107cache.*;
Cache cache;
try {
CacheFactory cacheFactory =
CacheManager.getInstance().getCacheFactory();
cache = cacheFactory.createCache(Collections.emptyMap());
} catch (CacheException e) {
// ...
}
Using jcache - get & putString key; // ...
byte[] value; // ...
// Put the value into the cache.
cache.put(key, value);
// Get the value from the cache.
value = (byte[]) cache.get(key);
Testing with ab benchmark
ab -c 50 -n 50 http://[app-address]/[servlet_path]
or
ab -c 50 -t 50 http://[app-address]/[servlet_path]
Other Operation● getAll(), putAll(), deleteAll()
A single read or write operation for multiple memcache entries
● increment(key, delta), incrementAll(...)Provide atomic increment of numeric value(s)
● getIdentifiable(), putIfUntouched()A mechanism to update a value consistently by concurrent requests
Memcache Caveat
● Memcache Is Volatile● Performance suggest: Batch size < 32 MB● Dedicated memcache (pay): 10K OPS/s/GB,
range 1-20 GB
Task Queue
Why Task Queue● Frontend restriction of 60 second limit● Need longer-running tasks● Asynchronously outside front end request● Best practice:
○ Email○ Write datastore○ Web crawler○ Backend jobs
Task Queue Types● By name
○ Default Queue (Push Queue)○ Named Queue
● By type○ Push queues
■ tasks execute automatically■ only for App Engine app
○ Pull queues■ tasks wait to be leased■ available to workers outside the app■ tasks can be batched
queue.xml<queue-entries>
<queue>
<name>Queue-Name</name>
<rate>1/s</rate>
...
</queue>
<queue>
<name>Queue-Name2</name>
<mode>pull</mode>
...
</queue>
</queue-entries>
Push Queue
Pull Queue
Task Queue Manage Console
Task Queue - Default Push Queue
Queue queue = QueueFactory.getDefaultQueue(); //named queue: //QueueFactory.getQueue("Qname");
TaskOptions task = TaskOptions.Builder .withUrl("/path-to-my-worker") .param(key, value);
queue.add(task);
Reachable Servlet
Detail Configure
● Task queues use token bucket algorithm [ref]
● rate - usual rate● bucket-size - cap on peak demand● max concurrent requests● Retry options
○ task-retry-limit - min retries○ task-age-limit - min elapsed time to keep retrying
(Both task-retry-limit and task-age-limit are reached, task is deleted)○ min-backoff-seconds -- min delay between retries○ max-backoff-seconds -- max delay between retries○ max-doublings -- max times to double delay
● more parameters see DOC
queue.xml parameters
Task Queue - With TransactionDatastoreService ds = DatastoreServiceFactory.getDatastoreService();Queue queue = QueueFactory.getDefaultQueue();try { Transaction txn = ds.beginTransaction();
// … other operations queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker")); // … other operations
txn.commit();} catch (DatastoreFailureException e) { // … exception handle}
Pull Queue - Create Queue
Queue queue =
QueueFactory.getQueue("queue_name");
TaskOptions task =
TaskOptions.Builder
.withMethod(TaskOptions.Method.PULL)
.payload(payload);
queue.add(task);
Pull Queue - Lease
Queue queue =
QueueFactory.getQueue("queue_name");
tasks = q.leaseTasks(
duration,
TimeUnit.SECONDS,
how-many);
Pull Queue - Delete Task
//delete named taskqueue.deleteTask("task_ name");
//purge all tasks from queueQueueFactory.getQueue("queue_name").purge();
Task Queue ACL - queue.xml
<queue>
<name>pull-queue1</name>
<mode>pull</mode>
<acl>
<user-email>[email protected]</user-email>
<writer-email>[email protected]</writer-email>
<writer-email>[email protected]</writer-email>
</acl>
</queue>
Cron
Cron
<cronentries>
<cron>
<url>/recache</url>
<description>
Put your description here
</description>
<schedule>every 5 minutes</schedule>
</cron>
</cronentries>
● every 12 hours● every 5 minutes from 10:00 to 14:00● 2nd,third mon,wed,thu of march 17:00● every monday 09:00● 1st monday of sep,oct,nov 17:00● every day 00:00● every N (hours|mins|minutes) ["from" (time) "to" (time)]● every 2 hours synchronized
Cron Schedule Format
Cron Manage Console
Restrictions
● 20 cron jobs for free apps● 100 for paid apps
Task Queue / Cron Reference● Task Queues Developer Guide (Java) (Python)
● Task Queue Configuration (Java) (Python)
● Task Queue API Ref (JavaDoc) (Python)
● Task Queue REST API Ref
● Scheduled Tasks with Cron (Java) (Python)
● Task queue quota and limits
● Get started with Task API article (Python)
● Google I/O 2011: Putting Task Queues to Work (video)