The DNA of a GREAT API

Ciprian Sorlea, CTOOctober 2015

Building API’s that developers will love

04.11.2015 2

What’s an API:

Any API:• allows Software components, systems and subsystems to interact.

• hides the complexity of the underlying architecture and functionality

• focuses on separation of concerns

• allows callers/consumers to access resources (ex. CRUD operations) or to perform various operations on top of the resources

04.11.2015 3

Qualities of a good API:

A good API must be:• domain specific, simple and easy to use• consistent (naming, parameters, etc.)• stateless• transparent & clear in terms of error trapping• accessible (from different platforms, languages, etc.)

04.11.2015 4

From Good To Great:

A great API must be:• well documented and should provide many working samples• fast, secure, scalable and reliable• forward and backwards compatible (versioning)• “self aware”: usage statistics, logs/traces

04.11.2015 5

From Good To Great:

A great API must go the extra mile

04.11.2015 6

Who are the consumers?

An API can be consumed by:- A web application- A desktop application - A mobile application- Another API- An integration test- A 3rd party integration- Internet Of Things- A home appliance, a wearable device, a smart gadget

04.11.2015 7

Now, let’s remember….

Who is your target audience?

Developers

04.11.2015 8

A big question & challenge:

How do we handle all these uses cases, while also making it easy for

Developers?

04.11.2015 9

Things to consider:

- Transport: how do we exchange data?- Resource models: what kind of data do we exchange?- API Architecture patterns: how do we design our API?- Error Handling: how do we handle errors & failure?- Testing: how do we make our API bullet proof?- Security: how do we protect our data (or ourselves)?- Manageability: how can we manage our API and support it operation-wise?- Evolution: how do we ensure our API is future-proof?- Adoption: how do we make sure our API will be adopted?

API Transporthow do we exchange data?

04.11.2015 11

API Transport

REST API:- Recommended for external integrations and applications, 3rd party apps and

plugins- Use JSON & XML over HTTP(s)- Use established Security mechanisms (OAuth2, Basic Authentication, etc.)

04.11.2015 12

API Transport

RPC / Binary Protocol:- Recommended for internal usage or high throughput requirements, IOT and

devices with limited networking capabilities / bandwidth- Use Apache Thrift, Google Protobuff or alternatives- Use simplified authentication and authorization model to minimize overhead

– assume working in safer environment

Note: For now, we’ll focus on

REST API’s

API Resource Modelswhat kind of data do we exchange?

04.11.2015 15

API Resource Models

Resources are the heart and soul of the API

Resources represent the abstraction of any information we exchange and operate on within an API

Domain Model ~= Resource Model

04.11.2015 16

API Resource Models

Resource classification:

- Singleton – can be part of a collection or can be stand-alone- ServerInfo – Server Information can be exposed as a standalone singleton resource- User – User information can be exposed as a resource, part of a collection

- Collection – a list of so called documents belonging to the same domain- Users- Products- Orders- Documents

04.11.2015 17

API Resource Models

Minimal Resource Model Attributes:

- URI: Unified Resource Identifier- represents the unique URL by which a resource can be accessed- HATEOAS takes this to the extreme level- Ex: http://api.domain.com/v1/serverInfo, http://api.domain.com/v1/users/1, http://

api.domain.com/v1/projects/

- ID:- required when the resource is a singleton as part of a collection- a numeric ID (ex: 1029) or a GUID (ex. c366728d-ca93-4b23-a4b9-ae360600b86a)- can be the database specific primary key (ex. auto generated) or can use a custom field with

random generated values, to enhance security

04.11.2015 18

API Resource Models

Important consideration:

- Ensure your resource models can be properly represented in any transport models (ex. JSON, XML) and without significant effort (Ex. Don’t try to use complicated bit level data in your models)

- Follow the Open Close principle. This will make your resources future-proof

- Ensure the API resource model doesn’t carry with it the DB optimizations and complexity

API Architecture Patterns(archetypes)

how do we design our API?

04.11.2015 20

API Archetypes

The de-facto API Archetypes:• Document – a singleton resource• Collection – a collection of resources• Repository (also known as Store) – a collection of immutable resources• Controller – an action dispatcher ***

*** Should be avoided. Explanation later.

04.11.2015 21

API Archetypes

Pseudo API Archetypes:• Queue – asynchronous processing of a long operation – can also be a batch operation

or a transaction, but not limited to those• Should provide status and progress information• Ex: File processing, order processing, etc.

• Transaction – multiple resource operations to be processed in a ALL-OR-NONE manner• The request fails if any of the operation fails.• Any simple action (see Controller) can be also seen/handled as a transaction

• Batch – a sequence of resource operations to be processed in sequence• Any failure is reported to the consumer, but doesn’t stop the processing of the batch • Should provide status and progress information• Ex. Batch create, batch update, etc.

04.11.2015 22

API Archetypes: Document

• Exposes functionality specific to a singleton resource.

• Some standalone documents (ex. ServerInfo), can be read-only

• Some collection-owned documents are mutable (can be updated, deleted, etc.) -> parent archetype is a Collection

04.11.2015 23

API Archetypes: Document – Partial Retrieval

Partial retrieval allows consumer to limit the transferred information. This leads to better performance and improved customer experience.

Limiting the fields:• Specify the fields to be included, as a query string parameter• Specify the fields to be ignored, as a query string parameter• Specify the fields to be expanded (sub-details), as a query string parameter• Specify a predefined view (predefined or custom definition of data retrieval

model)

04.11.2015 24

API Archetypes: Document – Partial Retrieval

Examples:http://api.domain.com/v1/users?fields=id,firstName,lastName,email,addresses(city)

http://api.domain.com/v1/users?excludedFields=email

http://api.domain.com/v1/users?view=minimalView

http://api.domain.com/v1/users?expand=addresses,previousEmails

04.11.2015 25

API Archetypes: Document – Partial Update

Allows consumers to update only those fields that need to be changedThis provides faster and better user experience.

Two ways:- Specifying the fields to be changed with the new value (partial documents)- Using operation maps

04.11.2015 26

API Archetypes: Document – Partial Update

Using partial documents:

PATCH /v1/users/1029 HTTP/1.1

Content-Type: application/json

X-API-PUBLIC-KEY: D6DB835A-35B1-42B3-8805-8F00F5344BB9

X-API-REQUEST-ID: 5BF75F7E-0A55-40DF-9E79-354B9B162971

{

“emailAddress”: ciprian.sorlea@nordlogic.com,

“companyName”: “NordLogic”

“phoneNumbers” : [“+4074ILOVEAPIS”]

}

04.11.2015 27

API Archetypes: Document – Partial Update

Advantages of using partial documents:- Request very clean and minimal- Requires very little extra effort to implement

Disadvantages:- Hard to add items to lists / arrays, remove values from maps, etc.

04.11.2015 28

API Archetypes: Document – Partial Update

Using operation maps:

PATCH /v1/users/1029 HTTP/1.1

Content-Type: application/json

X-API-PUBLIC-KEY: D6DB835A-35B1-42B3-8805-8F00F5344BB9

X-API-REQUEST-ID: 5BF75F7E-0A55-40DF-9E79-354B9B162971

{

“emailAddress”: { “set”: “Ciprian.Sorlea@nordlogic.com”},

“companyName”: { “set” : “NordLogic”},

“phoneNumbers” : { “add” : [“+4074ILOVEAPIS”] }

}

}

04.11.2015 29

API Archetypes: Document – Partial Update

Advantages of using operation maps :- More powerful control over what operations are made- Doesn’t require pre-fetching of documents prior to update to get the actual

map/list content

Disadvantages:- Requests tend to be more verbose, even though just slightly- More effort is needed to implement support for this approach (usually)

04.11.2015 30

API Archetypes: Document – Partial Update

PRO TIP: You can implement Partial Update with either approach, or you can combine

them and allow patching in either way.

04.11.2015 31

API Archetypes: Document – Partial Retrieval

PRO TIP: Combining Partial Update with Partial Retrieval, APIs can offer extreme

flexibility for a wide variety of consumers & devices.

04.11.2015 32

API Archetypes: Document – Partial Retrieval

PRO TIP: You can implement transactions for updating fields that require additional

details on change:

Ex. Change status + comment

04.11.2015 33

Document Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{documentID} or http://api.domain.com/{version}/{collection}/{documentID}

HTTP Verb DescriptionGET Retrieves the document (complete or partial)PUT Updates a document with the content provided. Only for Collections.PATCH Updates a document fields specified in the content, or applying the

specific update operations. Only for Collections.DELETE Deletes the document from the locationHEAD Verifies if the given document id exists. OPTIONS Lists the available operations / HTTP Verbs

04.11.2015 34

API Archetypes: Collection

Collections provide access to multiple document resourcesOperations are usually CRUD

Collection capabilities:- Pagination - Filtering: using simple filtering or a query language)- Sorting: simple (based on single field) or complex (based on multiple fields)- Partial field retrieval: identical to document level partial field retrieval

04.11.2015 35

API Archetypes: Collection Query String

Query String parameters are used to configure collection functionality (ex: sorting, filtering, etc.)Rules & recommendations:- Use same QS parameter names across your entire API, and document them

- Ensure you don’t introduce un-necessary constraints by choosing improper names

- As long as you don’t confuse your consumers, you can allow aliases for some parameters

Ex: - Page & PageSize work for a basic pagination but not for continuous scrolling – improper naming

- Start & limit work for any data pagination scenario

- Offset/From and count can be aliases to start & limit

04.11.2015 36

API Archetypes: Collection Query String

Pagination

Ex: http://api.domain.com/v1/users?from=100&count=10

http://api.domain.com/v1/users?count=10 (uses a default offset/start)

04.11.2015 37

API Archetypes: Collection Query String

Sorting- can be simple (single field + direction) or complex (list of fields, with specific

sort orders)- can be applied on 1st level fields or on details (expanded content)

Ex: http://api.domain.com/v1/users?sortBy=email,sortOrder=DESC

http://api.domain.com/v1/users?orderBy=name:ASC,email:DESC,contactInfo.phone

04.11.2015 38

API Archetypes: Collection Query String

Filtering:- Simple filters (all filters are of type EQUALS):

http://api.domain.com/v1/users?email=Ciprian.Sorlea@nordlogic.comhttp://api.domain.com/v1/users?companyName=Nordlogic&role=developer

- Advanced filters (specify the operation: EQ, NE, GT, LTE, etc.)http://api.domain.com/v1/users?companyName=LIKE:*logic&role=NE:qa

- Query (use a pseudo query language):http://api.domain.com/v1/users?query=companyName like ‘*logic’ and role not eq ‘QA’

http://api.domain.com/v1/users?query=name eq ‘John’ or name eq ‘Bill’

04.11.2015 39

Collection Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{collection}/

HTTP Verb DescriptionGET / Retrieves a list of paged, filtered and sorted documents, complete or

partialGET /{docID} Retrieves the document (complete or partial)POST / Creates a new document in the collection

HEAD Can be used to check if data exists given the provided filters and pagingDELETE Purges the collection. Recommended for Development only.OPTIONS Lists the available operations / HTTP Verbs

04.11.2015 40

Collection Archetype: Response

Few tips:

- Always echo the pagination details (ex. offset & count), to provide context

- Give more details about the rest of the data (is there more?). Ex: maxRows

- Don’t include empty/null fields and ensure (through documentation) your consumers handle that. (This applies to single documents as well)

04.11.2015 41

API Archetypes: Repository

Repositories provide access to multiple immutable document resourcesOperations are usually CRD (Create, Retrieve, Delete)Updates to documents within Repositories MIGHT happen behind the API level

Nice to have capabilities:- Capped repositories: limit data to most recent X records- Auto Expiration: automatically delete data older than X days/hours/minutes- Tailing options: retrieve latest X records

04.11.2015 42

Repository Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{repository}/

HTTP Verb DescriptionGET / Retrieves a list of documents with limited capabilities (ex. Tailing over

the records in an audit/trace log). Nov very common.GET /{docID} Retrieves the document (complete or partial).POST Creates a new document in the repositoryHEAD When GET is also implemented, this can be used to check if data exists.DELETE Purges the repository. Recommended for Development only.OPTIONS Lists the available operations / HTTP Verbs

04.11.2015 43

API Archetypes: Controller

Allows execution of application specific actions that cannot be mapped with any of the CRUD methods (specific to collections or repositories).

Should be avoided as much as possible. Actions should be reconsidered as one of the proposed pseudo-archetypes: batch, queue, transaction.

This would allow execution to be more manageable and traceable.

04.11.2015 44

Controller Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{param1}/{param2}/{controllerAction}

HTTP Verb DescriptionPOST Executes the specific action, with the given parameters

04.11.2015 45

API Archetypes: Queue

Allows execution of application specific actions (jobs) in an asynchronous way, while allowing proper management and traceability of the action.

Each action, once queued, has an identifier and a status. Therefore each action becomes a Resource.

Upon completion, the job document can embed the job result or can provide a link to the actual result.

04.11.2015 46

Queue Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{queue}/

HTTP Verb DescriptionGET /{id} Retrieves the queued job status, progress information and execution

result – if available.POST Submits a new job into the queue and retrieves the job identifier.DELETE /{id} Cancels a job (if the status allow the operation to be canceled)OPTIONS Lists the available operations / HTTP Verbs

04.11.2015 47

API Archetypes: Transaction

Allows execution of application specific actions in transactional manner.

Transactions can be asynchronous (see queues) or asynchronous.

In case of synchronous transactions, response contains the transaction result.

04.11.2015 48

Transaction Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{queue}/

HTTP Verb DescriptionGET /{id} Retrieves the transaction status, progress information and execution

result – if available.POST Submits a new transaction to be processed.DELETE /{id} Stops/cancels a transaction (in case of asynchronous/queued

transactions)OPTIONS Lists the available operations / HTTP Verbs

04.11.2015 49

API Archetypes: Batch

Allows execution of application specific actions over multiple records / bulk processing.

Batches must always be asynchronous, and the behavior is identical to Queues.

Progress information is mandatory in case of batches.

04.11.2015 50

Batch Archetype: URI & HTTP Verbs

• Base URI: http://api.domain.com/{version}/{batch}/

HTTP Verb DescriptionGET /{id} Retrieves the status, progress information and execution result – if

available.POST Submits a new batch job to be processed.DELETE /{id} Stops/cancels a batch jobOPTIONS Lists the available operations / HTTP Verbs

04.11.2015 51

Pseudo Archetypes

PRO TIP: You can combine Queues with Transactions and Batches in any way you

want.

Just make sure things make sense

04.11.2015 52

Pseudo Archetypes

PRO TIP: Sometimes you might want to have, for each Pseudo Archetype, a

validation equivalent.

Helps your applications/integrations cause less frustrations.

Error Handlinghow do we handle errors & failure?

04.11.2015 54

API Error Handling

Use specific HTTP Error Codes where appropriate for application logic specific errors. Ex:

- 401 Unauthorized: errors with authentication / authorization

- 404 Not Found: document does not exist

- ….

04.11.2015 55

API Error Handling

Response body should always contain:- Error code – what happened, coded for analytics

- Error message – human readable message, localized to user’s profile

- Error details – the context on the error.

Nice to have:- Error ID – id of the error, generated by the lowest level module where the error

occurred. Should also propagate and find itself in the log files.

- Flow ID – if possible, for cross-app tracking

- ….

04.11.2015 56

Pseudo Archetypes

PRO TIP: Certain operations can generate multiple errors (ex. multiple validation

errors). If applicable, provide a list of errors in the response instead of providing just the

first error encountered.

04.11.2015 57

Pseudo Archetypes

PRO TIP: Don’t provide too much details in the error messages. Consider privacy and

security before anything else.

Testinghow do we make our API bullet proof?

04.11.2015 59

API Testing

Option 1:

- Develop API Client along the actual API Service- Build Automated tests using the API Client, using the same programing

language

04.11.2015 60

API Testing

Option 2:

- Build Automated tests using a dedicated API Testing framework

• http://frisbyjs.com/• https://github.com/mkotsur/restito• https://github.com/rest-driver/rest-driver• https://github.com/dareid/chakram

04.11.2015 61

API Testing

Option 3:

- Use dedicated 3rd Party apps or Browser Plugins

• https://www.getpostman.com/• http://www.soapui.org/rest-testing/getting-started.html

Securityhow do we protect our data (or

ourselves)?

04.11.2015 63

Securing Rest API

Simple rules:

- Use disposable authentication tokens- Implement authentication mechanisms good for your target audience

Ex: OAuth2 works very well for web and mobile, but does it work for enterprise?

- Use granular roles & access rights for each resource model- Take the multi-tenancy into account from start- Use & require HTTPS whenever sensitive data is in transit- Encrypt any sensitive data at rest

04.11.2015 64

Securing Rest API

Preventive & Enterprise Ready Security:

- Audit any failed authentication / authorization- When using sessions, ensure the expiration time can be configured- Lock/disable authentication tokens after a configurable number of failed

authentication attempts- Use a whitelist / blacklist for request filtering- Implement overall rate limits as well as application / token specific limits- Validate any data coming IN. Never take things for granted.

04.11.2015 65

Securing Rest API

Preventive & Enterprise Ready Security:- Require (configurable) password strength- Use password expiration - Log Remote IP, User-Agent (and Application ID, if possible) with any request- Pass authentication tokens in the request headers, not in URL/Query String- Filter unaccepted query string parameters from URL’s- Escape/strip any textual information accepting user input

04.11.2015 66

Pseudo Archetypes

PRO TIP: NEVER, EVER, EVER, EVER

TRUST YOUR USERS.

Manageabilityhow can we manage our API and

support it operation-wise?

04.11.2015 68

Managing Rest API & Supporting Operations

Simple rules:- Log everything, almost everything.- Implement usage analytics and metrics- Time any operation, build performance metrics and alerts- Build operational dashboards, enable error notifications - Use change logs & audit trails for sensitive data- Require “4 eyes check” rule on any major change performed by admins- Use infrastructure as code

Evolutionhow do we ensure our API is future-

proof?

04.11.2015 70

Future proof your API

Simple rules:

- Version your resource models and archetypes from the beginning- Any new version of the resource model should extend the previous versions- Establish a support plan from first public release- Classify your API’s as public, private & unsupported to ensure phased

adoption

Adoptionhow do we make sure our API will be

adopted?

04.11.2015 72

Ensuring APIs adoption

- Make it easy to get started - developers should be 1 CURL request away from their first API call.

- Provide good and clear documentation.

- Provide SDK’s / client libraries and samples for most popular development platforms and languages of your target audience.

04.11.2015 73

Ensuring APIs adoption

- Ensure your business logic & rules don’t have to be guessed and duplicated by developers in their own apps (ex. validation rules).

- Eat your own dog food. Use your API to build your other products / integrations.

- Don’t introduce breaking changes. Plan to support your public API releases.

04.11.2015 74

Ensuring APIs adoption

PRO TIP: Deliver VALUE, not just JSON or XML.

Thanks for watching.