OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix

API ecosystem with Scala, Scalatra, and Swagger

at Netflix

Manish Pandit @lobster1234

Manish Pandit

Engineering Manager, Streaming Platforms

Netflix

@lobster1234

Thank you, OSCON!

50 M+ Streaming Members

40 countries and counting

Thousands of device types

Power the Consumer Electronics Partner Portal

Enable Certification of Netflix Ready Devices

Source of truth for all Device Data at Netflix

Correlate Streaming Quality Metrics

Our APIs

Devices

Smart devices +

Certification =

Lots of Device Metadata!

Model Firmware

Screen Resolution Subtitle Support

3D DRM

Remote Control Netflix SDK

4K HD …

Everything matters..

58 Resources

~10 methods per resource

Background Netflix OSS Components

Development Deployment/Delivery

Open Floor

Architecture

HTTP Layer, and Manager Layer

Cassandra

EVCache

Crowd/SSO

RDS

Astyanax

Netflix OSS Cloud Components

Manager Layer = Business Code

Protocol Independent

Package-able as an artifact

HTTP Layer = Protocol Wrapper

Map HTTP methods to Manager functions

Transform JSON <=> Case Classes

The road trip from your IDE to a Production Node

Yes, it can be fun!



Open Floor



Open Floor

Open Source Infrastructure Components

Libraries

Tools

Frameworks

PaaS with AWS IaaS



Open Floor

Scala



Open Floor

Immutable by default

Concise, yet expressive

Type inference

Multi paradigm

Java ecosystem

Functional

/** * Map results from a java.sql.ResultSet to a list of objects of another type * @param rs The result set * @param f Function that takes the result set and returns an instance of T * @tparam A The type * @return Sequence of instances of type A */ def mapResults[A](rs: ResultSet, f: ResultSet => A) : Seq[A] = { val array = ArrayBuffer[A]() while (rs.next) { array += f(rs) } array.toSeq }

/** * Function to return a cached value, or fetch it per the parameter f * @param key The key to look up in cache, or set against this key if not found * @param f The function that fetches the instance from some persistent store * @tparam A The type * @return None if not found in cache and persistent store, else Some[A] */ def withCache[A](key: String)(f: => Option[A]): Option[A] = {

EVCacheGateway.get[A](key) match { case None => val data = f if (!data.isEmpty) EVCacheGateway.set(key, data.get) data

case Some(x) => Some(x) } }

/** * Get a user given an ID * @param id The User ID * @return None if user is not found, else the user */ def getById(id: Long): Option[User] = {

withCache[User](s"User$id") { val query = "select * from users where user_id = ?" DBHelper.queryWith(query, extract, id).headOption

} }

Scalatra



Open Floor

Lightweight HTTP wrapper

.war based deployment

Swagger support

ScalatraSpec

/** * Get all the chipsets ordered by name. */get("/providers", operation(getChipsetProviders)) {

Ok( ChipsetManager.getProviders(0,20) ) }

ScalaTest



Open Floor



Open Floor

Simple

Intuitive

English-like

Rich

Promotes BDD

it should "return the chipset provider partner orgs" in { val data = ChipsetManager.getProviders(0,20) data.records.isEmpty should be (false) data.records.size should be (20) data.total should be >= 21l }

ScalatraSpec



Open Floor

Traits to take ScalaTest to the next level

Helpers for JSON parsing

Plenty of wrappers (body, headers..)

addServlet(new ChipsetService, "/*") it should "get a list of chipset providers" in { getWithToken("/providers")(TestUserTokens.testToken) { status should equal (200) body should include (""""start":0,"end":19,"count":20""") }}

Swagger



Open Floor

Excellent support within Scalatra

Type-safe documentation

Contract First API Sandbox

Machine Readable

get("/providers", operation(getAllChipsetProviders)) {//code

}case class PartnerList(data: Seq[PartnerOrg], start: Int, end: Int, count: Long, total: Long) def getAllChipsetProviders = authApiOperation[PartnerList]("getChipsetProviders", "Get all chipset providers") .parameter(queryParam[Option[Int]]("start").description("Starting record count, defaults to 0")) .parameter(queryParam[Option[Int]]("count").description("The number of records requested, defaults to 20"))

Swagger

Deployment



Open Floor

Push to dev

Jenkins runs dev build, tests, merges to

master

Jenkins runs master build, makes a .deb

Aminator bakes an AMI from

the .deb

Asgard deploys the AMI in

staging cloud

Production?



Open Floor

lgml-mpandit:nrd-portal-api mpandit$ git statusOn branch devYour branch is up-to-date with 'origin/dev'.Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory)

modified: api/src/main/scala/com/netflix/nrdportal/http/ChipsetService.scalamodified: api/src/test/scala/com/netflix/nrdportal/http/ChipsetServiceSpec.scalamodified: api/src/main/scala/com/netflix/nrdportal/manager/ChipsetManager.scalamodified: api/src/test/scala/com/netflix/nrdportal/manager/ChipsetManagerSpec.scala

no changes added to commit (use "git add" and/or "git commit -a")

Check in

[info] Passed: Total 1386, Failed 0, Errors 0, Passed 1386[success] Total time: 1273 s, completed Jul 19, 2014 8:39:54 PMBuild step 'Build using sbt' changed build result to SUCCESSPushing HEAD to branch master of origin repositoryReturning node parameter for ssh-dynaslave-1291624cTriggering a new build of PPD-NRD-PORTAL-API-MASTER #1172Notifying upstream projects of job completionFinished: SUCCESS

Dev Build



Open Floor

Integration Tests

Hitting Real DB

Refreshed every night with Production

Build Artifacts:nrd-portal-api_1.0-h1172.880e187_all.deb 127.71 MB[fingerprint]Changes:Added an endpoint to return a list of chipset provider partner orgs. NFLX-5514 (detail)

Started by upstream project PPD-NRD-PORTAL-API-DEV build number 1923originally caused by:Started by an SCM change

Revision: 880e1873fef63d278b3180b49af7434e234dee40origin/master

Master Build

Finally!

“A node, once deployed, cannot be changed.”

Immutable Deployments

Easy rollbacks

Consistency

Predictability

Immutable Deployments

Summary



Open Floor

Bridge the gap between dev and deploy

No such thing as too many tests

Automate everything

Document your APIs

Best solutions are implementation agnostic

Netflix OSS - http://netflix.github.io/#repo

Scalatra - http://www.scalatra.org/

Swagger - https://helloreverb.com/developers/swagger

ScalaTest - http://www.scalatest.org/

Session Feedback and Rating

slideshare.net/lobster1234

speakerdeck.com/mpandit

OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix

Technology

Transcript of OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix