Liliana Matos, Director, Front-End Engineering @ 1stdibs Rob Richard, Senior Director, Front-End Engineering @ 1stdibs

Defer and Stream Directives in GraphQLImprove Latency with Incremental Delivery

About

• Online marketplace for luxury items across multiple verticals

• Front-End stack: Node, GraphQL, React, and Relay

• Offices in New York, Vilnius, Lithuania, and Bangalore, India

What are @defer and @stream?

• @defer and @stream are proposed directives to the GraphQL Specification to support incremental delivery for state-less queries

• Championing since January 2020

• In collaboration with GraphQL Working Group

• In this talk, we will discuss:

• Motivation

• Specification proposal overview

• Code Examples

• Reference implementation in GraphQL.js

• Open-source contribution

• Best practices

query TalksQuery { talks(first: 6) @stream ( label: "talkStream", initialCount: 3 ) { name ...TalkComments @defer(label: "talkCommentsDefer") } }

fragment TalkComments on Talk { comments { body } }

Why @defer and @stream?

• Large datasets may suffer from latency

• All requested data may not be of equal importance

• Current options for applications to prioritize data, such as query splitting and pre-fetching, come with undesirable trade-offs

• @defer and @stream would allow GraphQL clients to communicate priority of requested data to the server without undesirable trade-offs

Query Splitting

• Fetch expensive/non-essential fields in a separate query after initial query

• Trade-offs:

• Increased latency for lower priority fields

• Client resource contention

• Increased server cost

/** Original Query */query SpeakerQuery($speakerId: String!) { speaker(speakerId: $speakerId) { name ...SpeakerPicture }}

fragment SpeakerPicture on Speaker { picture { height width url } }

/** Split Queries */query SpeakerInitialQuery($speakerId: String!) { speaker(speakerId: $speakerId) { name }}

query SpeakerFollowUpQuery($speakerId: String!) { speaker(speakerId: $speakerId) { ...SpeakerPicture }}

Pre-fetching

• Optimistically fetching data based on a prediction that a user will execute an action

• Trade-offs:

• Increased server cost due to incorrect predictions

https://www.apollographql.com/blog/introducing-defer-in-apollo-server-f6797c4e9d6e/

What about Subscriptions?

• Intention is for real-time and long connections

• @defer and @stream, intention is to lower latency for short-lived connections

Specification Proposal for @defer and @stream

@defer

• The @defer directive may be specified on a fragment spread.

• if: Boolean

• When true fragment may be deferred, if omitted defaults to true.

• label: String

• A unique label across all @defer and @stream directives in an operation.

directive @defer(if: Boolean, label: String) on FRAGMENT_SPREAD

@deferExample

query SpeakerQuery($speakerId: String!) { speaker(speakerId: $speakerId) { name ...SpeakerPicture @defer(label: “speakerPictureDefer”) }}

fragment SpeakerPicture on Speaker { picture { height width url } }

// Response Payloads

// Payload 1

{ "data": { "speaker": { "name": "Jesse Rosenberger" } }, "hasNext": true}

// Payload 2{

"label": "speakerPictureDefer", "path": ["speaker"], "data": { "picture": { "height": 200, "width": 200, "url": "jesse-headshot.jpg" } }, "hasNext": false}

@stream

• The @stream directive may be provided for a field of List

• if: Boolean

• When true fragment may be deferred, if omitted defaults to true.

• label: String

• A unique label across all @defer and @stream directives in an operation.

• initialCount: Int

• The number of list items the server should return as part of the initial response.

directive @stream(if: Boolean, label: String, initialCount: Int) on FIELD

@streamExample

query SpeakersQuery { speakers(first: 3) @stream(label: “speakerStream", initialCount: 1) { name }}

// Response Payloads

// Payload 1{ "data": { "speakers": [ { "name": "Jesse Rosenberger" } ] }, "hasNext": true}

// Payload 2{ "label": "speakerStream", “path": ["speakers", 1], "data": { "name": "Liliana Matos" }, "hasNext": true}

// Payload 3{ "label": "speakerStream", "path": ["speakers", 2], "data": { "name": "Rob Richard" }, "hasNext": false}

Response FormatOverview

• When an operation contains @defer or @stream directives, the GraphQL execution will return multiple payloads

• The first payload is the same shape as a standard GraphQL response

• Any fields that were only requested on a fragment that is deferred will not be present in this payload

• Any list fields that are streamed will only contain the initial list items

Response FormatDetails

• label — The string that was passed to the label argument of the @defer or @stream directive that corresponds to this results

• path — A list of keys from the root of the response to the insertion point

• hasNext — A boolean that is present and true when there are more payloads that will be sent for this operation. 

• data — The data that is being delivered incrementally.

• errors — An array of errors that occurred while executing deferred or streamed selection set

• extensions — For implementors to extend the protocol

Response FormatExample

query SpeakersQuery { speakers(first: 2) @stream(label: "speakerStream", initialCount: 1) { name ...SpeakerPicture @defer(label: "speakerPictureDefer") }}

fragment SpeakerPicture on Speaker { picture { url } }

// Response Payloads

// Payload 1{ "data": { "speakers": [ { "name": "Jesse Rosenberger" } ] }, "hasNext": true}

// Payload 2{ "label": "SpeakerPictureDefer", "path": ["speakers", 0, "picture"], "data": { "url": "jesse-headshot.jpg" }, "hasNext": true}

// Payload 3{ "label": "SpeakerStream", "path": ["speakers", 1], "data": { "name": "Liliana Matos" }, "hasNext": true}

// Payload 4{ "label": "SpeakerPictureDefer", "path": ["speakers", 1, "picture"], "data": { "url": "liliana-headshot.jpg" }, "hasNext": false}

How to use @defer and @stream in your GraphQL Server

• @defer - existing resolvers will work effectively

• @stream - need to consider how resolvers return data

Execution

fragment SpeakerPicture on Speaker { picture { height width url } }

{ "data": { "speaker": { "name": "Jesse Rosenberger" } }, "hasNext": true}

query SpeakerQuery($speakerId: String!) { speaker(speakerId: $speakerId) { name ...SpeakerPicture @defer(label: “speakerPictureDefer”) }}

{

"label": "speakerPictureDefer", "path": ["speaker"], "data": { "picture": { "height": 200, "width": 200, "url": "jesse-headshot.jpg" } }, "hasNext": false}

Fork execution to dispatcher

Initial Payload

Subsequent Payload

How to use @stream in your GraphQL Server

• Any List field can use the @stream directive

• What you return from your resolver matters

List return types in GraphQL-JS• GraphQL-JS supports returning several different data types in List resolvers.

• Array<T>, any Iterable, Promise<Iterable<T>>

• GraphQL engine will get all results at once

• Initial payload will be held up by this resolver

• Subsequent payloads will be sent immediately after

const resolvers = { Query: { items: async function (_, { filters }): Promise<Array<Item>> { const items = await api.getFilteredItems({ filters }); return items; }, },};

List return types in GraphQL-JS

• Array<Promise<T>>

• GraphQL engine will start waiting for all results

• Initial payload will be sent as soon as the "initialCount" values are ready

• Subsequent payloads will be sent as each promise resolves

• Requires knowing how many results there will be before the resolver returns

Returning Array<Promise<T>>

const resolvers = { Query: { items: async function (_, { filters }): Array<Promise<<Item>> { const itemIds = await api.filterItems({ filters }); return itemIds.map(async itemId => await api.getItemById(itemId)); }, },};

List return types in GraphQL-JS

• AsyncIterable, Async Generator function

• GraphQL engine will yield each result from the iterable

• Initial payload will be sent as soon as the "initialCount" values are ready

• Subsequent payloads will be sent as each new value is yielded

• Can determine asynchronously if the list is completed

Async Generator Function Resolverconst resolvers = { Query: { users: async function* (): AsyncIterable<User> { const db = new Database(); while (true) { // select one document const result = await db.getNext();

// end iteration if there are no documents returned if (!result) { break; }

yield result; }

return; }, },};

Server - Client Communication

• No websockets or any other stateful connection mechanism

• Works with common infrastructure and old browsers

• Spec is transport-agnostic, so you could use websockets

"Chunked transfer encoding (CTE) is a mechanism in which the encoder sends data to the player in a series of chunks. The player doesn’t have to wait until the complete segment is available"

transfer-encoding: chunked

Multipart HTTP

• Standard for encoding multiple payloads in a single HTTP request response

• Used for File Uploads to attach binary file data to form requests

• Used for Emails to add attachments to email body

• GraphQL Response can be encoded as multipart data of multiple JSON payloads

Multipart HTTP---Content-Type: application/jsonContent-Length: 590{ "data": { "talks": [{ "title": "Opening Keynote", "time": "10:30-11:00", "speaker": { "name": "Jesse Rosenberger" }, }, ...] }}---Content-Type: application/jsonContent-Length: 140{ "path": ["talks", 0, "comments"], "data": [{ "body": "Loved this!" }]}

query ConferenceQuery { talks { title time speaker { name } ...commentsFragment @defer }}

fragment commentsFragment on Talk { body}

GraphQL over HTTP

• GraphQL over HTTP is a proposed specification to define how GraphQL should be served over HTTP

• We have an RFC to add incremental delivery to this spec using chunked encoding and multipart responses

• https://github.com/graphql/graphql-over-http/pull/124

Server Codefunction sendPartialResponse( response: $Response, result: AsyncExecutionResult,): void { const json = JSON.stringify(result, null, 2); const chunk = Buffer.from(json, 'utf8'); const data = [ '', '---', 'Content-Type: application/json; charset=utf-8', 'Content-Length: ' + String(chunk.length), '', chunk, '', ].join('\r\n'); response.write(data);}// Close connectionresponse.end();

Client Code - Fetch

const response = await fetch(url, { method, headers, body })

// Don't call response.json()! // That waits for the whole response to finish loading.// const json = await response.json()

const reader = response.body.getReader();while (true) { const { value, done } = await reader.read(); if (done) return; handleChunk(value);}

Client Code - XMLHttpRequest

// This works in Internet Explorer 7!const xhr = new XMLHttpRequest();let index = 0;xhr.open(method, url);xhr.addEventListener("readystatechange", function onReadyStateChange() { const chunk = xhr.response.substr(index); handleChunk(chunk); index = xhr.responseText.length;});xhr.send(body);

Open Source Implementationfetch-multipart-graphql

• https://github.com/relay-tools/fetch-multipart-graphql

References

• Spec RFC: https://github.com/graphql/graphql-spec/blob/master/rfcs/DeferStream.md

• Spec Proposal: https://github.com/graphql/graphql-spec/pull/742

• GraphQL-JS: https://github.com/graphql/graphql-js/pull/2319

• express-graphql: https://github.com/graphql/express-graphql/pull/583

• fetch-multipart-graphql: https://github.com/relay-tools/fetch-multipart-graphql

• GraphQL over HTTP RFC: https://github.com/graphql/graphql-over-http/pull/124