Revealing HTTP Request and Response

Home / Revealing HTTP Request and Response

Revealing HTTP Request and Response

December 7, 2015 | Article | 1 Comment

In world wide web service, Hypertext Transfer Protocol (HTTP) is main application protocol used for communication and distributing information. Hypertext is a multi-linear set of objects, building a network by using logical links (thus called as hyperlinks) between the nodes (can be text or words).

A session on HTTP is actually a sequence of network request and response transactions. But what is these two things actually? In this article we will discuss about what request and response in HTTP is.

In this article I will also demonstrate the concept of HTTP request and response using two machine: my Slackware64 as a client and FreeBSD 8.3 with Apache as web server. The IP of client will be 192.168.1.5 and IP Of server will be 192.168.1.3. All scenario use an isolated network to ensure no noise occurred. Thus we only have 2 nodes connected peer to peer.

The Key Concept

HTTP is fall in as Application layer protocol (layer 7 in OSI reference model, or layer 4 in TCP/IP model). The default port is 80, except defined otherwise.

In network, at least there are two nodes communicate. One as the server and other as client. A client ask a request to server and a server must give a response.

HTTP is a stateless protocol, which means each and every connection is independent of each other. The atomic transaction is called as session which consists of one request and replied by one response.

HTTP is built on top of TCP (Transmission Control Protocol) in which established a connection between the server and a client. Despite of built on TCP, the stateless property of HTTP will not persist the connection after response is done given to client.

Peeking to the Network Level

To understand what data goes in and out when transaction is done, we will dive in to lower level.

In this scenario we will initiate connection from client to the server and capture all the traffic on the server that’s coming from the client and see what’s happening. Specifically we will use wget to download a file from web server. Therefore, we need at least two terminal on client: one for requesting using wget, one for tcpdump the network interface.

First, fire up tcpdump so we can get any data in and out from interface can be sniffed.

root[email protected]yvern:/# tcpdump -i eth0 -s0 -n -A host 192.168.1.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes

Next we use wget to send and receive data.

[email protected]:/# wget http://192.168.1.3
--2013-03-13 14:16:54--  http://192.168.1.3/
Connecting to 192.168.1.3:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45 [text/html]
Saving to: 'index.html'

100%[======================================>] 45          --.-K/s   in 0s

2013-03-13 14:16:54 (8.92 MB/s) - 'index.html' saved [45/45]

Now, let see what happende during the http request. Following is the result when we did connection:

14:16:54.185814 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [S], seq 3142401191, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 0,nop,wscale 7], length 0
E..<[email protected]@.*`...........P.M<..........0.........
. ..........
14:16:54.185834 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [S.], seq 618379695, ack 3142401192, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 2109181,nop,wscale 7], length 0
E..<[email protected]@.<..........P..$....M<......0.........
. ... ......
14:16:54.185856 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 0
E..4.[@[email protected]*g...........P.M<.$......V.(.....
. ... ..
14:16:54.185913 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [P.], seq 1:108, ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 107
E....\@[email protected])............P.M<.$......V.......
. ... ..GET / HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: 127.0.0.1
Connection: Keep-Alive

14:16:54.185948 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [.], ack 108, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 0
[email protected]@..t.........P..$....M=....V.(.....
. ... ..
14:16:54.225066 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [P.], seq 1:336, ack 108, win 342, options [nop,nop,TS val 2109220 ecr 2109181], length 335
[email protected]@..$.........P..$....M=....V.w.....
. /$. ..HTTP/1.1 200 OK
Date: Wed, 13 Mar 2013 07:16:54 GMT
Server: Apache/2.4.3 (Unix) PHP/5.4.7
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

<html><body><h1>It works!</h1></body></html>

14:16:54.225093 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 336, win 350, options [nop,nop,TS val 2109220 ecr 2109220], length 0
E..4.]@[email protected]*e...........P.M=.$......^.(.....
. /$. /$
14:16:54.225897 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [F.], seq 108, ack 336, win 350, options [nop,nop,TS val 2109221 ecr 2109220], length 0
E..4.^@[email protected]*d...........P.M=.$......^.(.....
. /%. /$
14:16:54.252701 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [F.], seq 336, ack 109, win 342, options [nop,nop,TS val 2109247 ecr 2109221], length 0
[email protected]@..r.........P..$....M=....V.(.....
. /?. /%
14:16:54.252738 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 337, win 350, options [nop,nop,TS val 2109247 ecr 2109247], length 0
[email protected]@.*c...........P.M=.$......^.(.....
. /?. /?

Well, lot of text we’ve got for a simple request and response. Let’s examine it in detail. On that scenario, we will divide to three stage for understand it easier.

Stage 1: Establishing TCP Connection

Like all of use know that on TCP based connection, we must establish a connection before communicate. This stage is called as three-way handshake. This “handshake” is done from client to server with three phase. Now if we examine closely, the first three connection is the handshake done by client and server. As seen here:

14:16:54.185814 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [S], seq 3142401191, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 0,nop,wscale 7], length 0
E..<[email protected]@.*`...........P.M<..........0.........
. ..........
14:16:54.185834 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [S.], seq 618379695, ack 3142401192, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 2109181,nop,wscale 7], length 0
E..<[email protected]@.<..........P..$....M<......0.........
. ... ......
14:16:54.185856 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 0
E..4.[@[email protected]*g...........P.M<.$......V.(.....

This article won’t cover three way handshake in detail. I assume you have known what and how three way handshake done.

At this stage we have established a connection between client and server. At the same point, client can now send a request to server. Therefore, let’s go to second stage.

Stage 2: Client Initiating HTTP GET Request

Next on our example, client will give a request to server. Client initiate a HTTP “GET” request.

GET is one of request defined for HTTP and used mostly on web. This request are used by clients for retrieving data from the server. If we look closely, it reveals some informations as below:

  1. What resource is requested by client. In our scenario, it’s a simple “/”.  GET “/” , which tell the server to retrieve the root directory (the default page of the website).
  2. What version of http is used. In our case its HTTP/1.1
  3. What agent used by client. In our case, we got information that the client use wget. More specific: Wget/1.14 (linux-gnu)
  4. What type of data client ready to accept. HTTP allows all MIME media types. In this scenario, the client is ready to accept any kind of data hence it is */*
  5. HOST field tells the hostname of the server, from which client is requesting the data.
  6. Most of clients request a keep-alive type of connection from the server. Keep alive is used to keep the tcp connection made by the client, so that the overhead of creating a tcp connection is reduced for subsequent requests. Despite of that, it is server who made the decision to keep TCP connection active or not. Note that although a keep alive connection made, HTTP is still a stateless protocol.

All can be seen here:

14:16:54.185913 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [P.], seq 1:108, ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 107
E....\@[email protected])............P.M<.$......V.......
. ... ..GET / HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: 127.0.0.1
Connection: Keep-Alive

Stage 3: Server Reply with HTTP Response

On getting the GET request from client, the server must responds, by revealing some information about itself and metadata about the data asked by the client along with the data.

  1. The server sends a status code (this informs client software about the result of the request). There are some common status code. Discussing all the status code is beyond the scope of this article thus we will limit ourselves to key point and discuss the rest on another article. The server replies with status code 200 and the http version it is using. In our case it is HTTP /1.1.
  2. We also got the date and time which response was originated.
  3. Server field tells us what server application is used. In this case is Apache.
  4. Last modified field tells us the modified date and time of the data requested.
  5. Etag is a string identifies the modification of the data requested. Mostly used for caching and improves the quality of web caching.
  6. Another important factor that makes http a wonderful protocol is that we can request your required bytes of a resource. For example we can download a file of 100M and in between the connection dropped we can later resume the download by specifying the range of bytes from where to start the download in the GET request. This method also used as foundation of download accelerator / manager like IDM does.
  7. The Content-Length field specifies the size of the resource requested in bytes. In our case it’s familiar Apache word: It works! and some formatting which takes 45 bytes.
  8. Connection: keep-alive. This conclude that our keep-alive request is granted by server. On some situation, where our keep-alive wish is not granted, we will get Connection: close.
  9. Content type specifies the type of the content. This file is simply a text with HTML format, which is Text/HTML.
  10. Last is the data after these headers.

We can see the actual packet from tcpdump output:

14:16:54.225066 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [P.], seq 1:336, ack 108, win 342, options [nop,nop,TS val 2109220 ecr 2109181], length 335
[email protected]@..$.........P..$....M=....V.w.....
. /$. ..HTTP/1.1 200 OK
Date: Wed, 13 Mar 2013 07:16:54 GMT
Server: Apache/2.4.3 (Unix) PHP/5.4.7
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

<html><body><h1>It works!</h1></body></html>

Client will give ACK to server and later after no more data transmitted, the connection will be terminated.

The HTTP Request Types

On above scenario, we only discuss about GET request type. In this section, we will get in touch with other requests:

HTTP Head

Similar to HTTP GET request. It is the easiest method to know the complete details of the resource available on a particular URL, without downloading the entire data.

In our scenario, if we use HEAD request we wil get all the header’s in the response without getting “<html><body><h1>It works!</h1></body></html>” message.

Mostly used to retrieve attributes / metada regardless of data and can save much of bandwidth if the data is big.

HTTP Post

Mostly used to send data from client to server. Following is the example of HTTP post request from client to server.

POST /receiver.php HTTP/1.1
Host: 192.168.1.3
User-Agent: ELinks/0.11.1 (textmode; Linux; 80x25-2)
Referer: http://192.168.1.3/
Accept: */*
Accept-Encoding: gzip
Accept-Language: en
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 62
name=sarath&last=pillai&email=&telephone=&comments=

As seen on above snippet, the request is being send to “receiver.php“. This data being send to the server will ocntain other header’s as well that we saw during our GET request example.

The last line sends the exact data to the server.

HTTP Put

Similar to the post request. PUT request sends or creates a resource in the specified URI. If the resource is already present in that specified URI, server then update the URI, otherwise the resource will be created.

HTTP Delete

Request server to delete a specific resource on a specified URI.

Well, it’s not advisable to configure a webserver for HTTP delete operation. However, if such functionality is desired we can use a HTTP Post operation using a web form which intern will delete a resource.

HTTP Trace

Used to troubleshoot HTTP webpages.

On simple case, if suppose a web page is not getting loaded the way we want in browser. We can then trace request is used to retrieve the complete request that the server got from the client back to the client itself. It is much like see what command you have send to server.

This request is mostly disabled in most of the web server’s. The reason is simple, this operation is similar to viewing web server log of the request we send.

Famous Response

Now, let’s discuss about some widely known HTTP status code. The list below is some status code often occurred on transaction. For a complete list, you can read it on this article.

400
Bad Request Type. The respond occured when the server recieves malformed request of any kind.
403
Forbidden Access. This response means we can get the resource as the resource has limited access and we don’t have the access.
404
The most famous error code. Not Found. This error indicates that the resource is not found on server side.
500
An unknown error occurred on web server. This error inform the client that some internal error happen on server side but the error is not specified or not a specific problem is shown by web server.

, ,

About Author

about author

xathrya

A man who is obsessed to low level technology.

1 Comment
  1. NodeJS HTTP - Xathrya.ID

    […] Every server should bind and listen to an arbitrary port. In our example, our server listen to port 4000. The server is handling an event request for HTTP server. This event is triggered when a client is request or connecting to our server. We set callback which has two arguments: request and response. […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Social media & sharing icons powered by UltimatelySocial