Building TweetEngine
Transcript of Building TweetEngine
![Page 2: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/2.jpg)
Goals of this talk
Using TweetEngine to explain key concepts of building cloud applicationsExplaining App Engine as a development environmentWhet your appetite!
![Page 3: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/3.jpg)
What is App Engine?
App Engine is a platform
You build & test your app
Then upload your app to Google
App Engine runs everything
No need to worry about machines, network, storage, scalability, etc.
![Page 4: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/4.jpg)
The king of ease-of-use
Extremely rapid developmentVery low barrier of entrySimple yet robust syntaxRich library of packages/modulesApp Engine's first language API
![Page 5: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/5.jpg)
Components
![Page 6: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/6.jpg)
Getting TweetEngine
Application: http://tweetengine.net/
Code: http://github.com/Arachnid/tweetengine
Prerequisites:Python 2.5buildoutlxml (for i18n features)
![Page 7: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/7.jpg)
OAuth Secure method for granting permissions to third-party integrationsAllows sites to use a trusted identity provider (Google, Twitter)"Valet key" for the internet
Image and "valet key" source: (http://hueniverse.com/oauth/guide/intro/)
![Page 8: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/8.jpg)
Standard OAuth flow (User)What does the User see?
1. Link to login with Twitter2. User is redirected to an http:
//twitter.com URL3. User logs in, grants access4. User is redirected back to site
![Page 9: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/9.jpg)
Standard OAuth flow (Server)What does the server do?
1. Acquire request token2. Generate login URL3. Redirect or link user to this
URL4. After user has logged in,
Twitter will redirect to a callback URL on server
5. Server saves token passed to callback URL
![Page 10: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/10.jpg)
TweetEngine + OAuth
OAuth request token saved in TwitterAccount modelSaved per Twitter Accountoauth.py handles signingapis.py handles proxied Twitter API calls
![Page 11: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/11.jpg)
Logging in to Twitter using OAuth
# src/tweetengine/add.pyauth_token = self.request.get("oauth_token")auth_verifier = self.request.get("oauth_verifier")user_info = client.get_user_info(auth_token, auth_verifier=auth_verifier)
# Create the twitter account account = model.TwitterAccount.get_or_insert( user_info["username"], oauth_token=user_info["token"], oauth_secret=user_info["secret"], name=user_info["name"], picture=user_info["picture"])
![Page 12: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/12.jpg)
Initializing an OAuth client
# src/oauth.py
OAuthClient.__init__(self, "twitter", consumer_key, consumer_secret, "http://twitter.com/oauth/request_token", "http://twitter.com/oauth/access_token", callback_url)
![Page 13: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/13.jpg)
Twitter API calls- Web handlers user AJAX- Backend calls go to TwitterApiHandler# src/tweetengine/handlers/apis.pyfor k,v in params.items(): if isinstance(v, unicode): params[k] = v.encode('utf8')
# Join all of the params together. params_str = "&".join(["%s=%s" % (encode(k), encode(params[k])) for k in sorted(params)]) # Join the entire message together per the OAuth specification. message = "&".join(["GET" if method == urlfetch.GET else "POST", encode(url), encode(params_str)])
# Create a HMAC-SHA1 signature of the message.key = "%s&%s" % (self.consumer_secret, secret)# Note compulsory "&".signature = hmac(key, message, sha1)digest_base64 = signature.digest().encode("base64").strip() params["oauth_signature"] = digest_base64 return urlencode(params)
![Page 14: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/14.jpg)
OAuth
The "de facto" standard for authentication on the web against a third party APITweetEngine's support can be easily generalized to be used on any OAuth supporting siteHighly recommend: New sites, do not build identity system, use OAuth or OpenID
![Page 15: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/15.jpg)
i18n - internationalization
Problem: maintaining localized versionsLocalization is more than translationsTweetEngine already supports English, German and Italian!
Image source: http://www.flickr.com/photos/natematias/22132919
![Page 16: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/16.jpg)
Workflow
1. Developer marks strings as requiring translation2. Translators work on .PO files3. These are compiled into efficient .POT files4. At runtime, translations are looked up when pages rendered
using "Accepts-Header"
![Page 17: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/17.jpg)
What does a .po file look like?
#. Default: "Collaborative tweeting"#: tweetengine/templates/base.pt:36msgid "subline"msgstr "Tweeting collaborativo"
#. Default: "Welcome!"#: tweetengine/templates/index.pt:3msgid "title-welcome"msgstr "Benvenuto!"
![Page 18: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/18.jpg)
Sample template code
tweetengine/templates/index.pt:
<metal:title fill-slot="title" i18n:translate="title-welcome">Welcome!</metal:title>
tweetengine/templates/base.pt:<h2><a href="/" i18n:translate="subline">Collaborative tweeting</a></h2>
Run from TweetEngine root: bin/i18nize
![Page 19: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/19.jpg)
src/tweetengine/handlers/base.py # Instantiate Chameleon template loaderfrom chameleon.zpt.loader import TemplateLoaderfrom tweetengine import model from tweetengine.menu import mainmenu
tpl_path = os.path.join(os.path.dirname(__file__), "..", "templates")tpl_loader = TemplateLoader(tpl_path)
# Later on in the file template_vars['target_language'] = self.request.headers.get('Accept-Language', None)tpl = tpl_loader.load('base.pt') template_vars['master'] = tpl.macros['master']tpl = tpl_loader.load('macros.pt')template_vars['macros'] = tpl.macrostpl = tpl_loader.load(template_file)self.response.out.write(tpl(**template_vars))
![Page 20: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/20.jpg)
Remember: i18n > translations
Resizing UI elements for languages with long/short wordsRight-to-left languages (Hebrew, Arabic) and layoutsTime and date formatsMetric/English systemMuch more ... this just a place to start!
![Page 21: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/21.jpg)
App Engine is also about tooling
Image source: http://www.flickr.com/photos/sparr0/4584964212/
![Page 22: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/22.jpg)
Two App Engine toolsAppStats
Application profile that is App Engine aware
Task QueuesBackground queueing and worker mechanism
Image source: http://www.flickr.com/photos/spbatt/3928384579/
![Page 23: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/23.jpg)
AppStatsTrack expensive requestsExpose bottlenecks in your codeShow you what code is being executed the most for "hotspot optimization"Run in production with minimal overhead
![Page 24: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/24.jpg)
AppStats example
![Page 25: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/25.jpg)
AppStats example (con't)
![Page 26: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/26.jpg)
How does it work?Uses Call Hooks
pre‐call hook records start .me, request, stackpost‐call hook records end .me, mcycles, response
Measuring times:real time using Python’s wall clock APIAPI megacycles using rpc.cpu_usage_mcyclesCPU megacycles using quota.get_request_cpu_usage()
Get stack contents from Python’s sys._getframe()Recording accumulates in memory objectWritten to memcache at end of each request
![Page 27: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/27.jpg)
AppStats for TweetEngine
Add AppStats as WSGI Middleware: app = recording.appstats_wsgi_middleware(app)
Add AppStats GUI to URL mapping (app.yaml): - url: /stats.* script: appstats/ui.py
Dev: http://localhost:8080/stats Prod (as admin): http://APPID.appspot.com/stats
That's it!
![Page 28: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/28.jpg)
Task Queues
Execute background tasksThrottle controlETACan be unique!Invoked using a web handler
![Page 29: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/29.jpg)
Standard async worker architecture
![Page 30: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/30.jpg)
Task Queues in App Engine# Creating a task # src/tweetengine/model.py taskqueue.Task(eta=send_time, name=task_name, url=twitter.ScheduledTweetHandler.URL_PATH).add()
# Task Handler # src/tweetengine/handlers/twitter.pyclass ScheduledTweetHandler(webapp.RequestHandler):
def post(self): publishApprovedTweets()
def publishApprovedTweets(): # Iterate through unsent OutgoingTweets and send them
![Page 31: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/31.jpg)
Task Queues in TweetEngine
Used to schedule TweetsETA parameter used to set an approximate execution timeTasks are named - this allows us to set times to check for OutgoingTweets that should be sent at this time and not duplicate work (we don't have to use cron)
![Page 32: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/32.jpg)
Task Queues
Allow developers to build asynchronous workers without hassle of building out queueing/polling systemHandles task uniqueness and transactional task creationCan be scheduled, throttled
![Page 33: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/33.jpg)
Summary: TweetEngine technologies
1. OAuth - "Valet key" for internet applications2. i18n - Using templates and message bundles in web
applications3. AppStats - App Engine request profiling toolkit4. Task Queues - Background work for App Engine
applications
![Page 34: Building TweetEngine](https://reader036.fdocuments.net/reader036/viewer/2022081403/555a8907d8b42a98568b52e8/html5/thumbnails/34.jpg)
Questions?
Application: http://tweetengine.net/
Code: http://github.com/Arachnid/tweetengine
[email protected]: @ikai