Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

@cczona

Full-Stack & Full CircleWhat The Heck Happens in an HTTP Request-Response Cycle

Carina C. Zonahttp://cczona.com/@cczona

Confident CodingMicrosoft San Francisco

October 20, 2012

Tuesday, October 23, 12When Estelle said she wanted to do a day of teaching "everything else" about web development, I knew right away that the HTTP request-response cycle is something I just _had_ to talk about.It's the topic that always gets skipped. "Welcome to web development! Let's start at HTML, CSS, and JavaScript!"We pass it right by. And we shouldn't.

@cczona

Hypertext Transfer Protocol (HTTP) is the core technology of the web

Tuesday, October 23, 12

Hypertext Transfer Protocol (HTTP) is THE CORE technology of the web.Yes, more essential than even HTML. Assets on the web can be HTML, but don't have to be. We download software, graphics, PDFs, Word docs, Flash...none of these is HTML. Nor is XML, or JSON. CSS, or JavaScript.Data flowing across the web relies either on HTTP, or its secure counterpart HTTPS.The web relies on the HTTP request-response cycle.

?

?


As a developer, you wants to take an interest in the cycle. It's where your work gets assembled. That assembly can be slow, or it can be fast. _Which_? Well, that depends on how the request-response cycle is managed.

As a developer, you want to _understand_ what the cycle passes through so that you can arrange it to do whatever you want.

Frontend

Backend


Web development is usually regarded as two parts: Frontend Backend

Client side

Server side


Sometimes we refer instead to Client side Server side

@cczona

Frontend & Backend refer to infrastructure

Client side & server side refer to programming languages


It's fine, we do it all the time. But technically imprecise: [slide]

Client side(JavaScript)

Server side(Python, Ruby, PHP,

Perl, Java, etc.)


So the cycle refers to the communication passing through our frontend & backend infrastructure. Later we'll look at how programming languages fit into this picture.


Let's add a User to this mental picture, too. Because someone's got to make the request, right? And someone's got to be interested in what response it gets. So now we've got at least 3 potential components. They even correspond pretty nicely to a common team configuration: a user experience designer, a frontend developer, a backend developer.

Website/app


Frontend and backend are different parts of a cohesive whole, the website. It's the user and site that are in relationship with each other, want something of each other, desiring to communicate with each other.

The frontend and backend _together_ form the stack of infrastructure for the site to do that.

Full stack


That's what we mean by "full stack". The combination of frontend and backend infrastructure, plus the technologies that allow them to interact.

request

response

HTTP/1.1 200 OK

GET /something


Voila. The request-response cycle just refers to passing a request message through the stack, and the stack delivering a response to that.

@cczona

Request


@cczona

URL


There's an important piece missing from that diagram so far, right? URL.

What is a URL anyway? Many people compare it to a street address. A series of progressive detail that narrow it down to an exact location.

@cczona

Anatomy of a URL


@cczona

URL Basics

http://cczona.com/ <scheme>://<host>/


http://cczona.com/ <scheme>://<host>/ At minimum, a URL needs two parts:

@cczona

URL Basics



Scheme - Browsers have lately been hiding the scheme from you. Don't be misled by that. Scheme is _mandatory_. It is an instruction, specifying which transfer method to use.

@cczona

URL Basics



Host - This part I _would_ compare to an address. It specifies where the host server is. We'll expand on that later.

@cczona

URL Common

http://cczona.com/blog/2012/10/20/confident-coding<scheme>://<host>/< path >


It usually also has at least one more part: Path - the name of a resource being requested

@cczona

URL Common

http://cczona.com/blog/2012/10/20/confident-coding<scheme>://<host>/< path > HOW WHERE WHAT


Consider these three parts the "How, Where, and What".

@cczona

URL All the parts!

http://carinazona:[email protected]:3000/blog;year=2012?search&term=Confident&month=October#session2

<scheme>://<user>:<password>@<host>:<port>/<path>;<params >?<query>#fragment


There are 6 optional additional parts available.

I know, crazy, right?

@cczona

URL Less Common

PortQueryFragment


Port - is where the host server is expected to be listening for an HTTP requestQuery - is one or more values intended to be used by an applicationFragment - is the name of some part within the requested resource.

@cczona

URL Rarely used for HTTP

UsernamePasswordParameters


@cczona

User-Agent


A request is submitted via a User-Agent.

@cczona

User-Agent examples

BrowserCrawlers/spiders/robotsScreen readersProgramsScripts


Browsers - are the most common by far. But many other user-agent types exist as well.

Crawlers/spiders/robots - automated site browser. Some examples include search engine indexers (Googlebot), archivers (Archive.org aka "The Internet Archive"), testing tools (curl-loader), monitoring services (uptime)

@cczona

User-Agent examples



Screenreaders - assistive devices that read content aloud

Programs - such as cURL & Wget

@cczona

User-Agent examples



Scripts - these can be written in any language capable of using the HTTP scheme. Which is pretty much any common one you can imagine. Perl, PHP, Ruby, and Python have HTTP libraries that make it easy to submit requests and accept responses.

@cczona

Method


The User-Agent submits the request using one of the defined HTTP methods.

@cczona

HTTP Methods

GETPOSTHEADPUTDELETEOPTIONSTRACE


GET - is the most common method. The request is communicated primarily via the URL itself

@cczona

HTTP Methods



POST - is also common, typically used via an HTML form action. The request is communicated via the URL and the form's data is communicated via the request's headers. GETs have a length limit, so POST is a handy alternative when your form might receive a whole lot of data.

@cczona

HTTP Methods



HEAD - the request is communicated the same as a GET, so via URL. But HEAD signals that the response should only contain headers, not body. Whereas the response to a GET is both.

PUT, DELETE, OPTIONS & TRACE have not been widely adopted.

@cczona

Headers


I've been mentioning headers. The obvious question is: _what the heck is a header_?

@cczona

Headers are metadata


A header is metadata about the request or response.

@cczona

Headers Firefox Web Developer Tools


Browsers' developer tools can display headers. Let's take a look at these.Notice the structure: each line is one key/value pair. The line starts with a key, then a colon, then the value.

@cczona

Headers Chrome Developer Tools


@cczona

Headers Safari Developer Tools


@cczona

Request headers

Include information about:

• request• user's environment• user-agent's capabilities• user-agent's identity*


Request headers originate at the user-agent. Its headers include information about: request user's environment user-agent's capabilities user-agent's identity*

@cczona

Request headers

Include information about:

• request• user's environment• user-agent's capabilities• user-agent's identity*


There's a huge caveat here. The "user-agent" header frequently is a lie. There's a long story as to why. Never, ever depend on the user-agent header to be accurate.

@cczona

Domain Name Server (DNS)


@cczona

DNS Distributed map of domains to IPs


When a request is submitted by the user, the user-agent first has to determine where to submit it. The URL's host part can be a domain name or an IP address. But the user-agent actually can only submit to an IP address. This is where DNS comes in. A domain name service (DNS) is a directory of domains and their IP addresses.

@cczona

Nameservers

For example: NS1.Dreamhost.com NS2.Dreamhost.com NS3.Dreamhost.com NS4.Dreamhost.com


If you've ever registered a domain name, you may have noticed that the domain was either configured with a default set of nameservers, or you had to provide the set of nameservers for your web hosting provider.

For example (slide)

@cczona

Nameservers

For example: NS1.dreamhost.com NS2.dreamhost.com NS3.dreamhost.com NS4.dreamhost.com


Nameservers are a secondary directory necessary for DNS resolution. They're responsible for keeping track of IP addresses where each of _your domain's_ services are located.

@cczona

Host Server


@cczona

Host Servers


The host server is not the web server. Rather, it's the hardware and operating system that house the web server, and potentially houses other infrastructure used by the site. So for example: OSX, Linux, Unix, Windows Server, or Novel NetWare.

request

response

HTTP/1.1 200 OK

GET /something


The host server passes the request off to the web server.

request

response

HTTP/1.1 200 OK

GET /something


@cczona

Web Server


@cczona

Web Server


A web server is software that's capable of taking an HTTP request as input, and of delivering an HTTP response as output.The most popular web servers are Apache, Microsoft IIS, and "engine X" (nginx).Let's take a look at the web server's log of our request.

@cczona

Web Server Request Log

108.75.73.197 - - [20/Oct/2012:13:58:09 +0000] "GET /blog/?search&term=Confident&month=October HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4"


Looks pretty much like gibberish, right? Ah, but wait....

@cczona

Web Server Request Log

108.75.73.197 - - [20/Oct/2012:13:58:09 +0000] "GET /blog/?search&term=Confident&month=October HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4"


Now see how many parts you can name already. What's the request method? The IP address? The path? And query string? Any thoughts on what that purple one is?

@cczona

Web Server Redirect


The web server interprets the request & determines how it should be handled.It looks at the request path and checks whether it matches any redirection directives.These may be temporary (e.g. "offline for maintenance"), or they may be permanent (e.g. "blog" is now "old-blog").

@cczona

Web Server Redirect


In lines 1 & 2, the web server gracefully handles files that have moved.In lines 4 & 5, it uses a regular expression to match any files that match the pattern.

@cczona

Web Server Redirect


Maybe you've seen something similar to this WordPress panel. It looks like some cool fancy feature. Naw. It's just using a web server redirect. Regular expressions are awesome.

@cczona

Web Server Redirect (example)


Here's the redirect. Every page and post of your WordPress blog is actually one script file + an HTTP redirect. That's it. It only look like infinite pages. Why do this? Flexibility, while enforcing consistency. You could change every URL without breaking a single link. Nice.

@cczona

Web Server Redirect (example)


Once a web server has finished doing redirects, it checks whether the new path matches a corresponding path in the web server's directory hierarchy. If it doesn't match, or there's a problem reading in the content, error time!

@cczona

404 Page Not FoundWeb Server


Aww, sad panda.Well, sad Octocat.

@cczona

Web Server 500 Internal Server Error


Uh-oh. Headache Puppy says badness happened.

@cczona

Response


When it does match, now the magic happens.

@cczona

HTML


The web server reads in whatever content the file provides. Text, html, image, output from a script...whatever.

request

response

HTTP/1.1 200 OK

GET /something


Passing the request down to a script gives us opportunity to generate some (or all) content dynamically. Often this involves using values passed in via the URL and/or POSTed data.

@cczona

Script


This example script generates output, starting with a response header (line 1), then declaring the end of headers (line 3), then outputting response body (starting at line 5).

request

response

HTTP/1.1 200 OK

GET /something


A script may also want to call on other services, such as a database.

@cczona

Example Response Response Message


Once the web server receives output suitable for delivery in an HTTP response, it send it on its way.

@cczona

Response headers

Mostly describe:• Content• How user-agent should handle it

First line is HTTP version and final status

HTTP/1.1 200 OK


The response likewise has headers. Most of these describe the content and how it should be handled by the user-agent.Notice that the first line of the response states the final status of the request. If things went well, that's "200 OK".

@cczona

Example Response Response Message


The user agent receives the response, and interprets it. In this case, the header had told the user-agent to interpret our response's body as HTML.

@cczona

Back to the Frontend


Now it's the user-agent's turn to process what it's received. It parses the content, to determine how to render it for the user. Cycle complete!

request

response

HTTP/1.1 200 OK

GET /something


Now it can determine what _other_ assets are needed. The user-agent fires off requests for those as well. JavaScript, CSS, images, etc. A bunch more cycles!

request

response

HTTP/1.1 200 OK

GET /something


When the user-agent has received everything it needs, its final steps are to layout the content, and render it for the user.

@cczona

Done.


Voila.

@cczona

Optimizations


So far we've seen that a pretty simple set of web infrastructure. But much more technology can optionally be involved.

@cczona

Optimizing

AJAXCachingExpires headersCompressionProxy serverContent delivery network (CDN)

MinificationSpritesDatabase indexes, operations, joinsPartialsREST/APIs


You may have noticed that there's a lot of potential here to make a site seem sluggish. So what we learn from the HTTP request-response cycle is that there are many opportunities to optimize for fast performance.

@cczona

Stay tuned...


Luckily, Estelle will be talking next about a bunch of methods for optimizing performance!

@cczona

Thank you!

Any questions?


Any questions for me?

@cczona

Credits

• Networking icons by http://icons.iconarchive.com

• Illustrations from "HTTP: The Definitive Guide" by David Gourley & Brian Totty

• Header screenshots from Safari Developer Tools, Chrome Developer Tools, and Firefox Web Developer Tools

• DNS diagram from HowStuffWorks.com

• 404 Page Not Found from http://www.github.com & htttp://mozilla.org

• 500 Internal Server from http://www.tumblr.com/tagged/internal-server-error


http://icons.iconarchive.com/

http://icons.iconarchive.com/

http://www.tumblr.com/tagged/internal-server-error

http://www.tumblr.com/tagged/internal-server-error

@cczona

Resources

• "HTTP: The Definitive Guide" by David Gourley & Brian Totty (O'Reilly)

• http://dev.opera.com/articles/view/http-basic-introduction/

• http://dev.opera.com/articles/view/http-lets-get-it-on/

• http://dev.opera.com/articles/view/http-response-codes/

• http://www.w3.org/community/webed/wiki/How_does_the_Internet_work

• http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

• http://en.wikipedia.org/wiki/Representational_state_transfer

• http://en.wikipedia.org/wiki/URI_scheme

• http://en.wikipedia.org/wiki/Ajax_(programming)


http://dev.opera.com/articles/view/http-basic-introduction/

http://dev.opera.com/articles/view/http-basic-introduction/

http://dev.opera.com/articles/view/http-lets-get-it-on/

http://dev.opera.com/articles/view/http-lets-get-it-on/

http://dev.opera.com/articles/view/http-response-codes/

http://dev.opera.com/articles/view/http-response-codes/

http://www.w3.org/community/webed/wiki/How_does_the_Internet_work#How_do_computers_communicate_via_the_Internet.3F

http://www.w3.org/community/webed/wiki/How_does_the_Internet_work#How_do_computers_communicate_via_the_Internet.3F

http://en.wikipedia.org/wiki/Representational_state_transfer

http://en.wikipedia.org/wiki/Representational_state_transfer

http://en.wikipedia.org/wiki/URI_scheme

http://en.wikipedia.org/wiki/URI_scheme

http://en.wikipedia.org/wiki/Ajax_(programming)

http://en.wikipedia.org/wiki/Ajax_(programming)

Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle

Documents

Transcript of Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle