Lessons Learned from Building a REST API on Google App Engine

Jonathan AltmanPresentation to GolangDC-October 29, 2015

Whitenoise Market Webapp

• White Noise by TMSoft (http://www.tmsoft.com/white-noise/) is the leading sleeping app for iOS, Android, Mac, and Windows

• Customer wanted a way to:

• Allow users to download additional content to the app

• Create a vibrant community for users to interact with each other

• Scale to the large demand of existing users

http://whitenoisemarket.com/

White Noise Market App

Project

• Build a RESTful API to drive Whitenoise Market’s web front-end

• Angular SPA front end, also built as part of the project

• User authentication with Google or Facebook account—OAuth2

• Role-based authorization

• Implied: customer will use the API from a native mobile client as well

• Golang on Google App Engine, leverage their APIs

Sample Calls

• GET /api/items — get all items

• GET /api/item/item_id — get data about the item with id item_id

GAE via Golang

• Project was approx. 6 person/weeks 2nd 1/2 2014, including front end

• Customer specification based on their research

• Inherited solid proof of concept app, but no firm API

• GAE golang support was still beta, long term support indeterminate

• Actual GAE API usage calls: outside the scope of this talk (but see https://cloud.google.com/appengine/docs/go/)

Issues

• Package management

• Routing

• REST response formulation/error logging

• OAuth2 support for providers other than Google

• Authorization

• Miscellaneous

Package Management

• goapp get not go get

• Not building an exe locally, packages need to be in source tree uploaded to GAE - feels weird compared to golang philosophy

Routing — GAE has choices

• Prefix hostname with module — exposing internals

• Dispatch file: dispatch.yaml — 10 routing rules max

• Roll your own — just start matching URLs in the main dispatch handler in your golang code

• or…

• and remember: Google Cloud Endpoints were not yet a thing. Probably the way to go today

Roll Your Own Router

3rd Party Router: Gorilla mux!

• http://www.gorillatoolkit.org/pkg/mux

• Gorilla web toolkit has a bunch of other nice parts

• Other 3rd party router libraries probably work fine

• Parameterization, method control

• GAE takes care of a lot of other things Gorilla toolkit provides

r.HandleFunc("/api/comments/{sid}",  handleGetComments).Methods("GET")r.HandleFunc(“/api/comments/{sid}",  aihttphelper.AuthenticatedEndpoint(HandleAddComment)).Methods("PUT")

REST Status/Response Logging

• Standard REST success and error responses

• gorca — https://github.com/icub3d/gorca

• gorca.LogAndMessage: Logs console message and returns short message plus status code

• gorca.WriteJSON: succesful responses

gorca.LogAndMessage(c,  w,  r,  err,  "error",  "not_authenticated",  http.StatusUnauthorized)  gorca.LogAndMessage(c,  w,  r,  err,  "error",  err.Error(),  http.StatusBadRequest)  gorca.WriteJSON(c,  w,  r,  map[string]interface{}{“status”:  "OK",  "tagAdded":  tagValue})  

OAuth2 Support - gomniauth

• GAE does OAuth2 authentication…only for Google

• gomniauth does OAuth2 authentication for multiple providers, including google (https://github.com/stretchr/gomniauth)

• jwt for HTTP Bearer Token — (https://github.com/dgrijalva/jwt-go)

• Accepted pull request in gomniauth allows setting http Transport used because the GAE runtime replaces net/http’s DefaultTransport with a context-based one https://github.com/stretchr/gomniauth/pull/23)

gomniauth Patch

• You have to fetch a Transport with the current requests’ GAE context, and pass that to gomniauth before doing authentication

• See https://github.com/jonathana/gomniauth/commit/3e2e23995b035e26bbd58a0f56cb2b2d61dbe993 for details/usage

Authorization

• Separate from authentication. What a user can do, once we know who the user is

• Wrapper function shown before:

• “Middleware” takes a target function with an extra argument beyond the normal HTTP request handler for the authenticated user information, and returns a normal HTTP handler function that does the authorization check and runs the target function if authorized

• Factory functions encapsulated role info, but could pass in ACL data

r.HandleFunc(“/api/comments/{sid}",  aihttphelper.AuthenticatedEndpoint(HandleAddComment)).Methods("PUT")

Authorization Middlewaretype  AiHandlerFunc  func(appengine.Context,  http.ResponseWriter,  *http.Request,  *aitypes.AIUserInfo)  func  generateAuthenticatedEndpoint(h  AiHandlerFunc,  requiredRoles  aitypes.RoleValue)  http.HandlerFunc  {   return  func(w  http.ResponseWriter,  r  *http.Request)  {     c  :=  appengine.NewContext(r)       authUser,  err  :=  AuthenticateRequest(c,  r)     if  (err  !=  nil)  {       gorca.LogAndFailed(c,  w,  r,  err)       return     }     //  401  User  not  authenticated     if  (authUser  ==  nil)  {       http.Error(w,  "",  http.StatusUnauthorized)       return     }     //  403  User  not  authorized  (authenticated,  but  no  permission  to  resource)     if  (requiredRoles  >  0  &&  !(hasRole(authUser,  requiredRoles))  {       http.Error(w,  "",  http.StatusForbidden)       return     }       //  User  is  authenticated  and  authorized     h(c,  w,  r,  authUser)   }}  func  AuthenticatedEndpoint(h  WnHandlerFunc)  http.HandlerFunc  {   return  generateAuthenticatedEndpoint(h,  0)}

Miscellaneous• Concurrency: ignored as a premature optimization. Issues with

urlfetch.Transport led to concern on runtime support/research time

• GAE API deprecation: not golang specific, but several APIs in use were deprecated post-project and had to be replaced (blobstore)

• GAE appears to be going to more of an a la carte model where existing components are replaced with general GCE equivalents

• Google Cloud Endpoints were not available at the time

Miscellaneous, cont.

• You’ll be playing with the JSON serialization properties. Javascript<->go naming rules mismatch: nobody wants Javascript properties to begin with capital letters. Also, I tend to prefer map[string]interface{} over defined structs where I can

• Using appengine.Context. You will need to, almost everywhere, whether it’s for working with datastore, making outbound http requests, or logging via its .Infof() call

Thank You!email: jonathan@async.io

github: jonathanatwitter: @async_io