Tag Archive : web

/ web

Before Wt: Installing Prerequisite for Wt

December 12, 2015 | Article | No Comments

In this article we will discuss about installation of various packages and software as requirement for Wt. The goal of this article is for everyone to successfully install prerequisites for building Wt.

For this purpose, I use Slackware64 14.0 as development machine. However, you can also use other machine.

This article can be broken down to two sections:

  • System requirements
  • Optional packages

System Requirements

Applications on this sections is the primary requirements for building Wt. At least there are two packages:

  • CMake: cross platform make configure utility.
  • Boost C++ Library: A general purpose library for C++. Latest version is recommended.

Another requirements are the connector support which depend on what would you like.

For FastCGI (Unix only)

  • Apache2 or other web server which supports the FastCGI protocol. You can go to this page for more information about Web Server (Apache, Lighttpd, Nginx, etc). When using Apache use mod_fastcgi-2.4.x or mod_fcgid-2.3.5 alternatively.
  • FastCGI development kit

For built in http daemon

For ISAPI (Windows only)

  • ISAPI, which only works for deploying Wt applications within Microsoft IIS server.

Optional Packages

The following packages are optional, but highly recommended for Wt. They enables additional features in Wt. The packages are:

  • OpenSSL: Enabling HTTPS protocol support for Client (Http::Client) and Server (Http::Server)
  • libharu: Enabling rendering support for PDF documents.
  • GraphicsMagick: Enabling rendering support for raster images like PNG or GIF
  • pango: Text rendering for TrueType font selection
  • PostgreSQL, MySQL/MariaDB, and Firebird: Enabling connectors database for ORM library (alongside the Sqlite3 connector). You can choose any (all whole).

CMake

Cross platform, open-source build system consists of tools designed to build, test, and packaging software. It used to control the software compilation process using simple platform and compiler independent configuration files.

For information on how to install CMake, follow this article.

Boost C++ Library

Portable C++ source libraries and work well with the C++ Standard Library. It is intended to be widely useful and usable across a broad spectrum of applications.

For information on how to install Boost Library, follow this article.

OpenSSL

libharu

Free, cross platform, open source library for generating PDF files.

For information on how to install libHaru, follow this article.

GraphicsMagick

The swiss army knife of image processing. It provides a robust and efficient collection of tools and libraries which support reading, writing, and manipulating an image in over 88 major formats including DPX, GIF, JPEG, JPEG-2000, PNG, PDF, PNM, and TIFF.

For information on how to install GraphicsMagick, follow this article.

pango

PostgreSQL

MySQL

MariaDB

Firebird

A relational database offering many ANSI SQL standard features that runs on various operating system. It offers excellent concurrency, high performance, and powerful language support for stored procedures and triggers.

For information on how to install FirebirdSQL, follow this article.

FastCGI Development Kit

whttpd

Build Apache HTTPD for Windows from Source

December 11, 2015 | Article | No Comments

Apache HTTP Server, commonly referred to as Apache, is a popular web server application. It is notably having a key role in initial growth of World Wide Web. Originally based on the NCSA HTTPd server, development of Apache began in early 1995 after work on the NCSA code stalled. Apache quickly overtook NCSA HTTPd as the dominant HTTP server, and has remained the most popular HTTP server in use since April 1996.

In this article we will bring ourselves to build Apache HTTP Server for Windows. Some material used here are:

    1. Windows 8, 64-bit
    2. Apache HTTPD 2.4.7 (latest per November 28th, 2013)
    3. Windows 8 Platform SDK, February 2003 or later
    4. Microsoft Visual Studio 2010
    5. Perl 5.16.3
    6. awk
    7. nasm 2.11.rc1

Also for material building the Apache

    1. apr, apr-util, apr-iconv
    2. PCRE (Perl Compatible Regular Expressions)
    3. zlib library (optional)
    4. OpenSSL libraries (optional)

You should also provide free disk space at least 200MB when compiling. After installation, Apache needs approximately 80 MB of disk space.

In this article, we will use Visual Studio Command Prompt instead of common Command Prompt.

Instead of compiling to x64 code, we will target the x86 architecture.

Opening Visual Studio Command Prompt

As stated before, we use Visual Studio Command Prompt instead of common Command Prompt. I use Visual Studio 2010 (Visual Studio 10) on Windows 8 64-bit. To activate Visual Studio Command Prompt, there are two methods: search & run Visual Studio Command Prompt from Start Screen, run Command Prompt then execute script to set environment variable.

If you want to use first method, make sure you use Visual Studio Command Prompt for amd64 or 64-bit instead of 32-bit.

If you want to do second method, you can do:

"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" amd64

Here we execute vcvarsall.bat and set the argument to amd64 to obtain necessary environment variables for the toolchain basically.

Grab the Materials

Source code of Apache HTTPD can be downloaded freely from here. Choose the closest mirror for you.

There are two Perl implementation for Windows: Strawberry Perl and ActiveState perl. I leave you to choose which one. Both can be downloaded from here.

Windows Software Developer Kit (SDK) for Windows 8 can be downloaded freely from here. Download the sdksetup.exe then run it. Choose Windows SDK from options available.

AWK is standard feature of most Unix-like operating system. For Windows, there is an alternative: Brian Kernighan’s http://www.cs.princeton.edu/~bwk/btl.mirror/ site has a compiled native Win32 binary, http://www.cs.princeton.edu/~bwk/btl.mirror/awk95.exe which you must save with the name awk.exe (rather than awk95.exe). It should be installed in environment path or known by Visual Studio.

NASM or Netwide Assembler is the assembler targeting x86 family processor. We can download nasm from it’s official site. The latest (version 2.11rc1 per November 28th, 2013) can be downloaded here. The one I use here is nasm-2.11rc1-installer.exe. Make sure nasm can be executed (the path is in the environment path).

APR (Apache Project Runtime) is used for building Apache. The project is a separated project from Apache HTTPD therefore we need to download it manually. Download it here http://apr.apache.org/download.cgi, select the appropriate mirror for you. The three we should download are: apr 1.5, apr-util 1.5.3, apr-iconv 1.2.1. Download the win32 version source code.

Perl Compatible Regular Expressions is regular expression pattern matching using the same syntax and semantics as Perl 5. It can be downloaded from www.pcre.org. The latest version is 8.33 which can be downloaded from ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/.

The zlib library is optional, used for mod_deflate. The current version is 1.2.8 and can be download from http://www.zlib.net/. In this case, the filename is zlib-1.2.8.tar.xz.

OpenSSL libraries is optional, used for mod_ssl and ab.exe with ssl support. You can obtain the OpenSSL for Windows from http://www.openssl.org/source/. Assuming we have downloaded it. In this case, the file name is openssl-1.0.1e.tar.gz.

Pre-Compilation Stage

In this article, I assume awk is installed as C:\Windows\awk.exe which should on the environment path.

Extract the Apache source code. Once it’s extracted we have “httpd-2.4.7” directory (for example: D:\httpd-2.4.7).

Extract the apr package to Apache’s srclib and rename them to apr, apr-iconv, apr-util respectively. Therefore we have three subdirectories apr, apr-iconv, and apr-util inside of “httpd-2.4.7/srclib”.

Extract the PCRE package to Apache’s srclib and rename it to pcre. Therefore we have “httpd-2.4.7/srclib/pcre”.

If you want to include zlib support, extract zlib source code inside Apache’s srclib sub directory and rename the directory to zlib. Therefore, we have “httpd-2.4.7/srclib/zlib”.

If you want to include openssl support, extract openssl source code inside Apache’s srclib sub directory and rename the directory to openssl. Therefore, we have “httpd-2.4.7/srclib/openssl”.

Compilation

The makefile script for Windows is defined as Makefile.win. In this article, we will build all the optional package first, manually.

To compile & build zlib, enter the Apache’s srclib for zlib and invoke the Makefile. Assuming Apache source code is in D:\httpd-2.4.7

cd D:\httpd-2.4.7\srclib\zlib
nmake -f win32\Makefile.msc AS=ml64 LOC="-DASMV -DASMINF -I." OBJA="inffasx64.obj gvmat64.obj inffas8664.obj"
nmake -f win32\Makefile.msc test
copy zlib.lib zlib1.lib

The last command is used to copy zlib.lib as zlib1.lib. We do this because when we compile OpenSSL we need library with this name.

To compile & build openssl, enter the Apache’s srclib for zlib and invoke the Makefile. Assuming Apache source code is in D:\httpd-2.4.7. To prepareOpenSSL to be linked to Apache mod_ssl or the abs.exe project, we can use following commands:

cd D:\httpd-2.4.7\srclib\openssl
perl Configure disable-idea enable-camellia enable-mdc2 enable-zlib VC-WIN64A -ID:\httpd-2.4.7\srclib\zlib -LD:\httpd-2.4.7\srclib\zlib
ms\do_win64a.bat
nmake -f ms\ntdll.mak

The above command configures OpenSSL with Visual C++ Win64 AMD. We disable the IDEA algorithm since this is by default disabled in the pre-distributions and really shouldn’t be missed. If you do however require this then go ahead and remove disable-idea.

Because we use OpenSSL 1.0.1e, we should invoke following command:

echo. > inc32\openssl\store.h

Now go to the top level directory of our Apache HTTPD source code. We will invoke the makefile to build Apache HTTPD.

nmake /f Makefile.win installr

Finally compile and install HTTPd, you can specify INSTDIR= to specify a path of where to install HTTPd as well. You can also specify database bindings by adding DBD_LIST=”mysql sqlite” etc. Also as it points out, don’t forget to add the libraries and includes from the databases to the INCLUDE and LIB

Installing Wt on Ubuntu

December 9, 2015 | Article | No Comments

Wt (pronounces as witty) is a C++ library for developing web application.

This article will discuss about installation of Wt on Ubuntu Linux, using package manager in specific. The prebuilt packages are made always updated to the newest Wt version so it would be the best for starting learning Wt.

Installing from the Official Package

Since Ubuntu Intrepid (8.10), official packages for Ubuntu are available. To install Wt, run:

aptitude install witty witty-dbg witty-dev witty-doc

This will automatically install all the required dependencies. If you only want the runtime library, you only need to install the witty package. The witty-dbg package contains the debug versions of the libraries. Make sure you install witty-dev (or libwtwhatever-dev) if you want to develop Wt applications.

The official package is usually a bit outdated due to the stabilization periods Debian and Ubuntu need prior to release. If you want to use the newest version of Wt and not build from source, read on.

Installing from Prebuilt Packages ghdg

Since Wt 2.0.3, unofficial packages for Ubuntu are being built by Pau Garcia i Quiles. To install Wt, add the Wt PPA to your repositories (check “Adding this PPA to your system” in the Wt PPA page). After adding the repository to your system, run:

 $ sudo apt-get update
 $ sudo aptitude install libwt*

This will automatically install all the required dependencies. If you only want the runtime library, you only need to install the libwt24, libwthttp24, libwtext24, etc packages (no -dev, -doc or -dbg packages). The libwt-dbg package contains the debug versions of the libraries.

These packages are built by the maintainer of the official Debian and Ubuntu packages and are always updated to the latest version of Wt. If you want to use the newest version of Wt and not build from source, this is the preferred method.

Please note in the past packages were named witty, witty-dev, witty-doc and witty-dbg. These package still exist but only as transitional packages and will be removed in the future. It is recommended that you install libwt*.

Wt C++ Web Toolkit Introduction

December 9, 2015 | Article | No Comments

Wt (pronounces as witty) is a C++ library for developing web application.

Wt offers abstraction of web-specific implementation details including client-server protocols, event handling, graphics support, graceful degradation (or progressive enhancement), and URL handling. The API is widget-centric, like Qt and other toolkits.

Unlike many page-base frameworks, Wt was designed for creating stateful application that are at the same time highly interactive (leveraging techniques such as WebSockets, and Ajax) and accessible (supporting plain HTML browsers), using automatic graceful degradation or progressive enhancement. Things that are natural and simple with Wt would require an impractical amount of effort otherwise: switching widgets using animations, while being perfectly indexed by search robots with clean URLs, or having a persistent chat widget open throughout, that even works in legacy browsers like Microsoft Internet Explorer 6.

Wt can acts as a stand alone Http(s)/WebSocket server or integrates through FastCGI with other web servers.

Why Wt

Page-based web frameworks (Django, Ruby on Rails, PHP, etc …) do not attempt to abstract underlying technologies (HTML/XHTML, JavaScript, CSS, Ajax, WebSockets, Comet, Forms, DHTML, SVG/VML/Canvas). As a consequence, a web developer needs to be familiar with all of these evolving technologies and is also responsible for graceful degradation when browser support is lacking. The structure of many web applications still follows mostly the page-centric paradigm of early day HTML. This means that not only will you need to implement a controller to indicate how a user moves from page to page, but when using advanced Ajax or WebSockets, you will need to design and maintain your client-server communication.

Pure Ajax frameworks on the other hand require tedious JavaScript programming to deal with browser quirks, and client-server programming to interact securely with server resources. These applications usually are not compliant with accessibility guidelines and cannot be indexed by a search robot.

Generating HTML code or filling HTML templates is prone to security problems such as XSS (Cross-Site-Scripting) by unwillingly allowing JavaScript to be inserted in the page, and CSRF (Cross-Site Request Forgery) by trusting cookies for authentication. These security problems are hard to avoid in traditional frameworks when as a developer you need to implement JavaScript functionality and thus the framework cannot filter it out.

In contrast, a web application developed with Wt is developed against a C++ API, and the library provides the necessary HTML, CSS, Javascript, CGI, SVG/VML/Canvas and Ajax code. The responsibility of writing secure and browser-portable web applications is carried by Wt. For example, if available, Wt will maximally use JavaScript, Ajax and even WebSockets, but applications developed using Wt will also function correctly when JavaScript is not available. Wt will start with a plain HTML/CGI application and progressively enhance to a rich Ajax application if possible. With Wt, security is built-in and by default.

Typical Use Scenario

  • High performance, complex web applications which are fully personalized (and thus cannot benefit from caching), fully Ajax enabled and at the same time entirely accessible and Search Engine Optimized.
  • Web-based GUIs for embedded systems benefit from the low footprint of a C++ web application server.
  • Web-based GUIs that require integration with (existing) C++ libraries, for example for scientific or engineering applications, or existing C++ desktop applications.

Other Benefits

  • Develop web applications using familiar desktop GUI patterns.
  • Provides an extensive set of widgets, which work regardless of JavaScript availability (but benefit from JavaScript availability).
  • A single specification for both client- and server-side validation and event handling.
  • Optionally, use XHTML and CSS for layout and decoration.
  • Generates standards compliant HTML or XHTML code.
  • Portable, anti-aliased graphics optimized for web usage (using inline VML, inline SVG, HTML5 canvas or PNG images), which can also be used to render to PDF.
  • Avoid common security problems since Wt has complete control over the presentation layer and proactively filters out active tags and attributes, does not expose business logic, and simplifies authentication using a stateful design.
  • Ultra-fast load time and low bandwidth usage, which are affected only by screen complexity, not application size. Wt implements all the common tips and tricks for optimizing application responsiveness and even optimizes per browser.
  • A simple API with a robust cross-browser implementation for server-initiated events aka server push (using comet or WebSockets).
  • Use the built-in httpd for easy development and deployment, or use the FastCGI/ISAPI connectors to deploy in existing web servers.

Features

Core Library

  • Supports major browsers (Firefox/Gecko, Internet Explorer, Safari, Chrome, Konqueror, and Opera) but also plain HTML browsers (Links, Lynx).
  • Develop and deploy on Unix/GNU Linux or Microsoft Windows (Visual Studio) environments.
  • Equal behavior with or without support for JavaScript or Ajax, as far as possible, by using graceful degradation or progressive enhancement.
  • Integrated Unicode support and pervasive localization.
  • Efficient rendering and (very) low latency.
  • Support for browser history navigation (back/forward buttons and bookmarks), pretty URLs with HTML5 History if available, and search engine optimization with a unified behavior for plain HTML or Ajax sessions.
  • Configurable session tracking options that include URL rewriting and cookies.
  • High performance, allowing deployment on low-end embedded devices, or energy-, space- and budget-friendly deployment of Internet or extranet web sites.
  • Completely based on event-driven async I/O: sessions are not tied to threads, and neither do open connections block threads. Instead, threads are needed only to improve concurrent request handling or for reentrant event loops.

Event Handling

  • Uses a modern typesafe signal/slot API for responding to events.
  • Listen for keyboard, mouse and focus events, and get event details (such as mouse position, modifier buttons, or keyboard key).
  • Automatic synchronization of form field data between browser and server.
  • Dynamic C++-to-JavaScript translation, by specifying stateless slot implementations. A single C++ slot implementation provides both client-side and server-side event handling: visual changes at client-side and application state at server side.
  • Possibility to hook in custom JavaScript (e.g. for client-side only event handling), and emit C++ signals from this custom JavaScript.
  • Drag&Drop API.
  • Timed events and server-initiated updates (“server push”)
  • Uses plain HTML CGI, Ajax or WebSockets

Native Painting Support

  • Unified painting API which uses the browsers native (vector) graphics support (inline VML, inline SVG, or HTML5 canvas), or renders to common image formats (PNG, GIF, …) or vector formats (SVG, PDF).
  • Supports arbitrary painter paths, clipping, text, images, transformations, drop shadow.

GUI Component

GUI component is composed of various widgets. For comprehensive examples, you can visit this link.

Built-in Security

  • Kernel-level memory protection protects against privacy issues arising from programming bugs, since sessions can be completely isolated from each other (in dedicated-process mode).
  • Supports encryption and server authentication using Secure Sockets Layer (SSL) or Transport Layer Security (TLS) through HTTPS.
  • Enables continuous use of HTTPS through low bandwidth requirements (fine-grained Ajax).
  • Built-in Cross-Site Scripting (XSS) prevention. Rendered text is always filtered against potentially malicious code, making XSS attacks against Wt applications (close to) impossible.
  • Not vulnerable to Cross-site Request Forgery (CSRF) because cookies for session tracking are optional, and even when used, they are never solely relied on for requests that trigger event handling code.
  • Not vulnerable to breaking the application logic by skipping to a particular URL, since only those events exposed in the interface can be triggered.
  • Session hijacking mitigation and risk prevention
  • DoS mitigation
  • A built-in authentication module implements best practices for authentication, and supports third party identity providers using OAuth 2.0, and (later) OpenID Connect

Object Relational Mapping Library

Wt comes with Wt::Dbo, a self-contained library which implements Object-Relational mapping, and thus a convenient way to interact with SQL databases from C++. Although features like optimistic concurrency control make this an ideal technology for a database driven web application (and it provides good integration with Wt’s MVC classes), the library can also be used for other applications, and does not depend on Wt. The ORM library has the following features:

  • No code generation, no macro hacks, no XML configuration, just modern C++!
  • Uses a templated visitor pattern which requires a single template method to provide the mapping: DRY and as efficient as conceivable!
  • You can indicate surrogate auto-incremental keys or map natural keys of any C++ type, which may also be composite (i.e. require more than one database field).
  • Supports optimistic concurrency control using a version field.
  • Maps Many-to-One and Many-to-Many relations to STL-compatible collections.
  • Provides schema generation (aka DDL: data definition language) and CRUD operations (aka DML: data manipulation language).
  • Prepared statements throughout.
  • Each session tracks dirty objects and provides a first-level cache.
  • Flexible querying which can query individual fields, objects, or tuples of any these (using Boost.Tuple).
  • Use a single connection or share connection pools between multiple sessions from which connections are used only during an active transaction.
  • Comes with Sqlite3, Firebird, MariaDB/MySQL and PostgreSQL backends, and an Oracle backend is also available on request.

Deployment

Wt library abstracts different deployment options as connectors libraries, which connect Wt with the outer world. Switching deployment option is a matter of relinking to one of these connector libraries.

Built-in httpd

  • Simple, high-performance web application server (multi-threaded, asynchronous I/O) based on the C++ asio library.
  • Supports the HTTP(S) and WebSocket(S) protocols.
  • Supports response chunking and compression.
  • Single process (convenient for development and debugging), and embeddable in an existing application.
  • Supports deployment behind a ProxyPass’ing (and if needed, load-balancing) web server.
  • Available for both UNIX and Win32 platforms.

FastCGI

  • Integrates with most common web servers (apache, lighttpd).
  • Different session-to-process mapping strategies.
  • Hot deployment: new sessions use the new application version while older sessions may continue with their application version.
  • Available only for UNIX platforms.

ISAPI

  • Integrates with Microsoft IIS server.
  • Uses the ISAPI asynchronous API for maximum performance.
  • Available for the Win32 platform.

Revealing HTTP Request and Response

December 7, 2015 | Article | 1 Comment

In world wide web service, Hypertext Transfer Protocol (HTTP) is main application protocol used for communication and distributing information. Hypertext is a multi-linear set of objects, building a network by using logical links (thus called as hyperlinks) between the nodes (can be text or words).

A session on HTTP is actually a sequence of network request and response transactions. But what is these two things actually? In this article we will discuss about what request and response in HTTP is.

In this article I will also demonstrate the concept of HTTP request and response using two machine: my Slackware64 as a client and FreeBSD 8.3 with Apache as web server. The IP of client will be 192.168.1.5 and IP Of server will be 192.168.1.3. All scenario use an isolated network to ensure no noise occurred. Thus we only have 2 nodes connected peer to peer.

The Key Concept

HTTP is fall in as Application layer protocol (layer 7 in OSI reference model, or layer 4 in TCP/IP model). The default port is 80, except defined otherwise.

In network, at least there are two nodes communicate. One as the server and other as client. A client ask a request to server and a server must give a response.

HTTP is a stateless protocol, which means each and every connection is independent of each other. The atomic transaction is called as session which consists of one request and replied by one response.

HTTP is built on top of TCP (Transmission Control Protocol) in which established a connection between the server and a client. Despite of built on TCP, the stateless property of HTTP will not persist the connection after response is done given to client.

Peeking to the Network Level

To understand what data goes in and out when transaction is done, we will dive in to lower level.

In this scenario we will initiate connection from client to the server and capture all the traffic on the server that’s coming from the client and see what’s happening. Specifically we will use wget to download a file from web server. Therefore, we need at least two terminal on client: one for requesting using wget, one for tcpdump the network interface.

First, fire up tcpdump so we can get any data in and out from interface can be sniffed.

[email protected]:/# tcpdump -i eth0 -s0 -n -A host 192.168.1.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes

Next we use wget to send and receive data.

[email protected]:/# wget http://192.168.1.3
--2013-03-13 14:16:54--  http://192.168.1.3/
Connecting to 192.168.1.3:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45 [text/html]
Saving to: 'index.html'

100%[======================================>] 45          --.-K/s   in 0s

2013-03-13 14:16:54 (8.92 MB/s) - 'index.html' saved [45/45]

Now, let see what happende during the http request. Following is the result when we did connection:

14:16:54.185814 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [S], seq 3142401191, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 0,nop,wscale 7], length 0
E..<[email protected]@.*`...........P.M<..........0.........
. ..........
14:16:54.185834 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [S.], seq 618379695, ack 3142401192, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 2109181,nop,wscale 7], length 0
E..<[email protected]@.<..........P..$....M<......0.........
. ... ......
14:16:54.185856 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 0
E..4.[@[email protected]*g...........P.M<.$......V.(.....
. ... ..
14:16:54.185913 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [P.], seq 1:108, ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 107
E....\@[email protected])............P.M<.$......V.......
. ... ..GET / HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: 127.0.0.1
Connection: Keep-Alive

14:16:54.185948 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [.], ack 108, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 0
[email protected]@..t.........P..$....M=....V.(.....
. ... ..
14:16:54.225066 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [P.], seq 1:336, ack 108, win 342, options [nop,nop,TS val 2109220 ecr 2109181], length 335
[email protected]@..$.........P..$....M=....V.w.....
. /$. ..HTTP/1.1 200 OK
Date: Wed, 13 Mar 2013 07:16:54 GMT
Server: Apache/2.4.3 (Unix) PHP/5.4.7
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

<html><body><h1>It works!</h1></body></html>

14:16:54.225093 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 336, win 350, options [nop,nop,TS val 2109220 ecr 2109220], length 0
E..4.]@[email protected]*e...........P.M=.$......^.(.....
. /$. /$
14:16:54.225897 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [F.], seq 108, ack 336, win 350, options [nop,nop,TS val 2109221 ecr 2109220], length 0
E..4.^@[email protected]*d...........P.M=.$......^.(.....
. /%. /$
14:16:54.252701 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [F.], seq 336, ack 109, win 342, options [nop,nop,TS val 2109247 ecr 2109221], length 0
[email protected]@..r.........P..$....M=....V.(.....
. /?. /%
14:16:54.252738 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 337, win 350, options [nop,nop,TS val 2109247 ecr 2109247], length 0
[email protected]@.*c...........P.M=.$......^.(.....
. /?. /?

Well, lot of text we’ve got for a simple request and response. Let’s examine it in detail. On that scenario, we will divide to three stage for understand it easier.

Stage 1: Establishing TCP Connection

Like all of use know that on TCP based connection, we must establish a connection before communicate. This stage is called as three-way handshake. This “handshake” is done from client to server with three phase. Now if we examine closely, the first three connection is the handshake done by client and server. As seen here:

14:16:54.185814 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [S], seq 3142401191, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 0,nop,wscale 7], length 0
E..<[email protected]@.*`...........P.M<..........0.........
. ..........
14:16:54.185834 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [S.], seq 618379695, ack 3142401192, win 43690, options [mss 65495,sackOK,TS val 2109181 ecr 2109181,nop,wscale 7], length 0
E..<[email protected]@.<..........P..$....M<......0.........
. ... ......
14:16:54.185856 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [.], ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 0
E..4.[@[email protected]*g...........P.M<.$......V.(.....

This article won’t cover three way handshake in detail. I assume you have known what and how three way handshake done.

At this stage we have established a connection between client and server. At the same point, client can now send a request to server. Therefore, let’s go to second stage.

Stage 2: Client Initiating HTTP GET Request

Next on our example, client will give a request to server. Client initiate a HTTP “GET” request.

GET is one of request defined for HTTP and used mostly on web. This request are used by clients for retrieving data from the server. If we look closely, it reveals some informations as below:

  1. What resource is requested by client. In our scenario, it’s a simple “/”.  GET “/” , which tell the server to retrieve the root directory (the default page of the website).
  2. What version of http is used. In our case its HTTP/1.1
  3. What agent used by client. In our case, we got information that the client use wget. More specific: Wget/1.14 (linux-gnu)
  4. What type of data client ready to accept. HTTP allows all MIME media types. In this scenario, the client is ready to accept any kind of data hence it is */*
  5. HOST field tells the hostname of the server, from which client is requesting the data.
  6. Most of clients request a keep-alive type of connection from the server. Keep alive is used to keep the tcp connection made by the client, so that the overhead of creating a tcp connection is reduced for subsequent requests. Despite of that, it is server who made the decision to keep TCP connection active or not. Note that although a keep alive connection made, HTTP is still a stateless protocol.

All can be seen here:

14:16:54.185913 IP 192.168.1.5.48262 > 192.168.1.3.80: Flags [P.], seq 1:108, ack 1, win 342, options [nop,nop,TS val 2109181 ecr 2109181], length 107
E....\@[email protected])............P.M<.$......V.......
. ... ..GET / HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: 127.0.0.1
Connection: Keep-Alive

Stage 3: Server Reply with HTTP Response

On getting the GET request from client, the server must responds, by revealing some information about itself and metadata about the data asked by the client along with the data.

  1. The server sends a status code (this informs client software about the result of the request). There are some common status code. Discussing all the status code is beyond the scope of this article thus we will limit ourselves to key point and discuss the rest on another article. The server replies with status code 200 and the http version it is using. In our case it is HTTP /1.1.
  2. We also got the date and time which response was originated.
  3. Server field tells us what server application is used. In this case is Apache.
  4. Last modified field tells us the modified date and time of the data requested.
  5. Etag is a string identifies the modification of the data requested. Mostly used for caching and improves the quality of web caching.
  6. Another important factor that makes http a wonderful protocol is that we can request your required bytes of a resource. For example we can download a file of 100M and in between the connection dropped we can later resume the download by specifying the range of bytes from where to start the download in the GET request. This method also used as foundation of download accelerator / manager like IDM does.
  7. The Content-Length field specifies the size of the resource requested in bytes. In our case it’s familiar Apache word: It works! and some formatting which takes 45 bytes.
  8. Connection: keep-alive. This conclude that our keep-alive request is granted by server. On some situation, where our keep-alive wish is not granted, we will get Connection: close.
  9. Content type specifies the type of the content. This file is simply a text with HTML format, which is Text/HTML.
  10. Last is the data after these headers.

We can see the actual packet from tcpdump output:

14:16:54.225066 IP 192.168.1.3.80 > 192.168.1.5.48262: Flags [P.], seq 1:336, ack 108, win 342, options [nop,nop,TS val 2109220 ecr 2109181], length 335
[email protected]@..$.........P..$....M=....V.w.....
. /$. ..HTTP/1.1 200 OK
Date: Wed, 13 Mar 2013 07:16:54 GMT
Server: Apache/2.4.3 (Unix) PHP/5.4.7
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Content-Length: 45
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

<html><body><h1>It works!</h1></body></html>

Client will give ACK to server and later after no more data transmitted, the connection will be terminated.

The HTTP Request Types

On above scenario, we only discuss about GET request type. In this section, we will get in touch with other requests:

HTTP Head

Similar to HTTP GET request. It is the easiest method to know the complete details of the resource available on a particular URL, without downloading the entire data.

In our scenario, if we use HEAD request we wil get all the header’s in the response without getting “<html><body><h1>It works!</h1></body></html>” message.

Mostly used to retrieve attributes / metada regardless of data and can save much of bandwidth if the data is big.

HTTP Post

Mostly used to send data from client to server. Following is the example of HTTP post request from client to server.

POST /receiver.php HTTP/1.1
Host: 192.168.1.3
User-Agent: ELinks/0.11.1 (textmode; Linux; 80x25-2)
Referer: http://192.168.1.3/
Accept: */*
Accept-Encoding: gzip
Accept-Language: en
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 62
name=sarath&last=pillai&email=&telephone=&comments=

As seen on above snippet, the request is being send to “receiver.php“. This data being send to the server will ocntain other header’s as well that we saw during our GET request example.

The last line sends the exact data to the server.

HTTP Put

Similar to the post request. PUT request sends or creates a resource in the specified URI. If the resource is already present in that specified URI, server then update the URI, otherwise the resource will be created.

HTTP Delete

Request server to delete a specific resource on a specified URI.

Well, it’s not advisable to configure a webserver for HTTP delete operation. However, if such functionality is desired we can use a HTTP Post operation using a web form which intern will delete a resource.

HTTP Trace

Used to troubleshoot HTTP webpages.

On simple case, if suppose a web page is not getting loaded the way we want in browser. We can then trace request is used to retrieve the complete request that the server got from the client back to the client itself. It is much like see what command you have send to server.

This request is mostly disabled in most of the web server’s. The reason is simple, this operation is similar to viewing web server log of the request we send.

Famous Response

Now, let’s discuss about some widely known HTTP status code. The list below is some status code often occurred on transaction. For a complete list, you can read it on this article.

400
Bad Request Type. The respond occured when the server recieves malformed request of any kind.
403
Forbidden Access. This response means we can get the resource as the resource has limited access and we don’t have the access.
404
The most famous error code. Not Found. This error indicates that the resource is not found on server side.
500
An unknown error occurred on web server. This error inform the client that some internal error happen on server side but the error is not specified or not a specific problem is shown by web server.

List of HTTP Status Code

December 7, 2015 | Article | 1 Comment

HTTP Status codes is one of data sent from server to client as part of response. This response indicate what is the status of client request on server side. All of the statuses are defined by Internet Engineering Task Force (IETF) using some related Request For Comments (RFC) documents.

Currently, the official registry of HTTP status code are maintained by Internet Assigned Numbers Authority (IANA).

Note that, some server also extend the codes with their status codes. This is not universally implemented but on this article we will list all of the known and reported status codes used as response.

The Taxonomy

HTTP status code is consists of three digit number range from 0 to 9 for each digit. The first digit indicated what category of message is and the rest of digit indicates what specific information it has. Globally, the status code is divided into five categories with first digit range from 1 to 5.

1xx Informational

This category indicates that request has been received and process is continuing. This class of status is provisional response, consisting of the status-line and optional headers and terminated by an empty line. The status code is not defined on HTTP/1.0

100 Continue
This means that the server has received the request headers, and that the client should proceed to send the request body (in the case of a request for which a body needs to be sent; for example, a POST request). If the request body is large, sending it to a server when a request has already been rejected based upon inappropriate headers is inefficient. To have a server check if the request could be accepted based on the request’s headers alone, a client must send Expect: 100-continue as a header in its initial request and check if a 100 Continue status code is received in response before continuing (or receive 417 Expectation Failed and not continue).
101 Switching Protocols
This means the requester has asked the server to switch protocols and the server is acknowledging that it will do so.[2]
102 Processing (WebDAV; RFC 2518)
As a WebDAV request may contain many sub-requests involving file operations, it may take a long time to complete the request. This code indicates that the server has received and is processing the request, but no response is available yet.[3] This prevents the client from timing out and assuming the request was lost.

2xx Success

The information is received and the request is processed successfully. In specifically the server has received, understood, accepted, and process the request.

200 OK
Most common response given by server. This is the standard response for successful HTTP requests. The actual response will depend on the request method used. In a GET request, the response will contain an entity corresponding to the requested resource. In a POST request the response will contain an entity describing or containing the result of the action.
201 Created
The request has been fulfilled and resulted in a new resource being created.
202 Accepted
The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place.
203 Non-Authoritative Information (since HTTP/1.1)
The server successfully processed the request, but is returning information that may be from another source.
204 No Content
The server successfully processed the request, but is not returning any content.
205 Reset Content
The server successfully processed the request, but is not returning any content. Unlike a 204 response, this response requires that the requester reset the document view.
206 Partial Content
The server is delivering only part of the resource due to a range header sent by the client. The range header is used by tools like wget to enable resuming of interrupted downloads, or split a download into multiple simultaneous streams.
207 Multi-Status
The message body that follows is an XML message and can contain a number of separate response codes, depending on how many sub-requests were made.
208 Already Reported (WebDAV; RFC 5842)
The members of a DAV binding have already been enumerated in a previous reply to this request, and are not being included again.
250 Low on Storage Space (RTSP; RFC 2326)
The server returns this warning after receiving a RECORD request that it may not be able to fulfill completely due to insufficient storage space. If possible, the server should use the Range header to indicate what time period it may still be able to record. Since other processes on the server may be consuming storage space simultaneously, a client should take this only as an estimate.
226 IM Used (RFC 3229)
The server has fulfilled a GET request for the resource, and the response is a representation of the result of one or more instance-manipulations applied to the current instance.

3xx Redirection

The request is received, but client must take additional action to complete the request. In this class, the further action must be performed by clients.The action required may be carried out by the user agent without user interaction and only if the method used in the second request is GET or HEAD. A user agent should not automatically redirect a request more than five times, since such redirections usually indicate an infinite loop.

300 Multiple Choices
Indicates multiple options for the resource that the client may follow. It, for instance, could be used to present different format options for video, list files with different extensions, or word sense disambiguation.
301 Moved Permanently
This and all future requests should be directed to the given URI.
302 Found
This is an example of industry practice contradicting the standard. The HTTP/1.0 specification (RFC 1945) required the client to perform a temporary redirect (the original describing phrase was “Moved Temporarily”), but popular browsers implemented 302 with the functionality of a 303 See Other. Therefore, HTTP/1.1 added status codes 303 and 307 to distinguish between the two behaviours. However, some Web applications and frameworks use the 302 status code as if it were the 303.
303 See Other (since HTTP/1.1)
The response to the request can be found under another URI using a GET method. When received in response to a POST (or PUT/DELETE), it should be assumed that the server has received the data and the redirect should be issued with a separate GET message.
304 Not Modified
Indicates that the resource has not been modified since the version specified by the request headers If-Modified-Since or If-Match. This means that there is no need to retransmit the resource, since the client still has a previously-downloaded copy.
305 Use Proxy (since HTTP/1.1)
The requested resource is only available through a proxy, whose address is provided in the response. Many HTTP clients do not correctly handle responses with this status code, primarily for security reasons.
306 Switch Proxy
No longer used.Originally meant “Subsequent requests should use the specified proxy.”
307 Temporary Redirect (since HTTP/1.1)
In this case, the request should be repeated with another URI; however, future requests should still use the original URI.In contrast to how 302 was historically implemented, the request method is not allowed to be changed when reissuing the original request. For instance, a POST request should be repeated using another POST request.
308 Permanent Redirect (experimental RFC)
The request, and all future requests should be repeated using another URI. 307 and 308 (as proposed) parallel the behaviours of 302 and 301, but do not allow the HTTP method to change. So, for example, submitting a form to a permanently redirected resource may continue smoothly.

4xx Client Error

Request is received, but server indicate that the request is error on client side. In this case, server should include an entity containing an explanation of the error situation, and explain whether it is a temporary or permanent condition. From RFC, the user agent should display any included entity to the user.

400 Bad Request
The request cannot be fulfilled due to bad syntax.
401 Unauthorized
Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource.
402 Payment Required
Reserved for future use. The original intention was that this code might be used as part of some form of digital cash or micropayment scheme, but that has not happened, and this code is not usually used.
403 Forbidden
The request was a valid request, but the server is refusing to respond to it. Unlike a 401 Unauthorized response, authenticating will make no difference. On servers where authentication is required, this commonly means that the provided credentials were successfully authenticated but that the credentials still do not grant the client permission to access the resource (e.g. a recognized user attempting to access restricted content).
404 Not Found
One of the most response found. The requested resource could not be found but may be available again in the future. Subsequent requests by the client are permissible.
405 Method Not Allowed
A request was made of a resource using a request method not supported by that resource; for example, using GET on a form which requires data to be presented via POST, or using PUT on a read-only resource.
406 Not Acceptable
The requested resource is only capable of generating content not acceptable according to the Accept headers sent in the request.
407 Proxy Authentication Required
The client must first authenticate itself with the proxy.
408 Request Timeout
The server timed out waiting for the request.According to W3 HTTP specifications: “The client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at any later time.”
409 Conflict
Indicates that the request could not be processed because of conflict in the request, such as an edit conflict.
410 Gone
Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed and the resource should be purged. Upon receiving a 410 status code, the client should not request the resource again in the future. Clients such as search engines should remove the resource from their indices. Most use cases do not require clients and search engines to purge the resource, and a “404 Not Found” may be used instead.
411 Length Required
The request did not specify the length of its content, which is required by the requested resource.
412 Precondition Failed
The server does not meet one of the preconditions that the requester put on the request.
413 Request Entity Too Large
The request is larger than the server is willing or able to process.
414 Request-URI Too Long
The URI provided was too long for the server to process.
415 Unsupported Media Type
The request entity has a media type which the server or resource does not support. For example, the client uploads an image as image/svg+xml, but the server requires that images use a different format.
416 Requested Range Not Satisfiable
The client has asked for a portion of the file, but the server cannot supply that portion. For example, if the client asked for a part of the file that lies beyond the end of the file.
417 Expectation Failed
The server cannot meet the requirements of the Expect request-header field.
418 I’m a teapot (RFC 2324)
Well, this is not an actual code used by HTTP server. Originally on 1998, it is used by IETF as Appril Fools’ jokes.
420 Enhance Your Calm (Twitter)
Also not HTTP standard. This code is returned by the Twitter Search and Trends API when the client is being rate limited. Other services may wish to implement the 429 Too Many Requests response code instead.
422 Unprocessable Entity (WebDAV; RFC 4918)
The request was well-formed but was unable to be followed due to semantic errors.
423 Locked (WebDAV; RFC 4918)
The resource that is being accessed is locked.
424 Failed Dependency (WebDAV; RFC 4918)
The request failed due to failure of a previous request (e.g. a PROPPATCH).
424 Method Failure (WebDAV)
Indicates the method was not executed on a particular resource within its scope because some part of the method’s execution failed causing the entire method to be aborted.
425 Unordered Collection (Internet draft)
Defined in drafts of “WebDAV Advanced Collections Protocol”, but not present in “Web Distributed Authoring and Versioning (WebDAV) Ordered Collections Protocol”.
426 Upgrade Required (RFC 2817)
The client should switch to a different protocol such as TLS/1.0.
428 Precondition Required (RFC 6585)
The origin server requires the request to be conditional. Intended to prevent “the ‘lost update’ problem, where a client GETs a resource’s state, modifies it, and PUTs it back to the server, when meanwhile a third party has modified the state on the server, leading to a conflict.”
429 Too Many Requests (RFC 6585)
The user has sent too many requests in a given amount of time. Intended for use with rate limiting schemes.
431 Request Header Fields Too Large (RFC 6585)
The server is unwilling to process the request because either an individual header field, or all the header fields collectively, are too large.
444 No Response (Nginx)
Used in Nginx logs to indicate that the server has returned no information to the client and closed the connection (useful as a deterrent for malware).
449 Retry With (Microsoft)
A Microsoft extension. The request should be retried after performing the appropriate action.
Often search-engines or custom applications will ignore required parameters. Where no default action is appropriate, the Aviongoo website sends a “HTTP/1.1 449 Retry with valid parameters: param1, param2, . . .” response. The applications may choose to learn, or not.
450 Blocked by Windows Parental Controls (Microsoft)
A Microsoft extension. This error is given when Windows Parental Controls are turned on and are blocking access to the given webpage.
451 Parameter Not Understood (RTSP)
The recipient of the request does not support one or more parameters contained in the request.
451 Unavailable For Legal Reasons (Internet draft)
Defined in the internet draft “A New HTTP Status Code for Legally-restricted Resources”. Intended to be used when resource access is denied for legal reasons, e.g. censorship or government-mandated blocked access. A reference to the 1953 dystopian novel Fahrenheit 451, where books are outlawed.
451 Redirect (Microsoft)
Used in Exchange ActiveSync if there either is a more efficient server to use or the server can’t access the users’ mailbox.
The client is supposed to re-run the HTTP Autodiscovery protocol to find a better suited server.
452 Conference Not Found (RTSP)
The conference indicated by a Conference header field is unknown to the media server.
453 Not Enough Bandwidth (RTSP)
The request was refused because there was insufficient bandwidth. This may, for example, be the result of a resource reservation failure.
454 Session Not Found (RTSP)
The RTSP session identifier in the Session header is missing, invalid, or has timed out.
455 Method Not Valid in This State (RTSP)
The client or server cannot process this request in its current state. The response SHOULD contain an Allow header to make error recovery easier.
456 Header Field Not Valid for Resource (RTSP)
The server could not act on a required request header. For example, if PLAY contains the Range header field but the stream does not allow seeking.
457 Invalid Range (RTSP)
The Range value given is out of bounds, e.g., beyond the end of the presentation.
458 Parameter Is Read-Only (RTSP)
The parameter to be set by SET_PARAMETER can be read but not modified.
459 Aggregate Operation Not Allowed (RTSP)
The requested method may not be applied on the URL in question since it is an aggregate (presentation) URL. The method may be applied on a stream URL.
460 Only Aggregate Operation Allowed (RTSP)
The requested method may not be applied on the URL in question since it is not an aggregate (presentation) URL. The method may be applied on the presentation URL.
461 Unsupported Transport (RTSP)
The Transport field did not contain a supported transport specification.
462 Destination Unreachable (RTSP)
The data transmission channel could not be established because the client address could not be reached. This error will most likely be the result of a client attempt to place an invalid Destination parameter in the Transport field.
494 Request Header Too Large (Nginx)
Nginx internal code similar to 431 but it was introduced earlier.
495 Cert Error (Nginx)
Nginx internal code used when SSL client certificate error occurred to distinguish it from 4XX in a log and an error page redirection.
496 No Cert (Nginx)
Nginx internal code used when client didn’t provide certificate to distinguish it from 4XX in a log and an error page redirection.
497 HTTP to HTTPS (Nginx)
Nginx internal code used for the plain HTTP requests that are sent to HTTPS port to distinguish it from 4XX in a log and an error page redirection.
499 Client Closed Request (Nginx)
Used in Nginx logs to indicate when the connection has been closed by client while the server is still processing its request, making server unable to send a status code back.

5xx Server Error

Request is received but server failed to fulfill an apparently valid request. In this case, the server cannot give the appropriate response due to some error occurred on server side.

500 Internal Server Error
A generic error message, given when no more specific message is suitable.
501 Not Implemented
The server either does not recognize the request method, or it lacks the ability to fulfill the request.
502 Bad Gateway
The server was acting as a gateway or proxy and received an invalid response from the upstream server.
503 Service Unavailable
The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.
504 Gateway Timeout
The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.
505 HTTP Version Not Supported
The server does not support the HTTP protocol version used in the request.
506 Variant Also Negotiates (RFC 2295)
Transparent content negotiation for the request results in a circular reference.
507 Insufficient Storage (WebDAV; RFC 4918)
The server is unable to store the representation needed to complete the request.
508 Loop Detected (WebDAV; RFC 5842)
The server detected an infinite loop while processing the request (sent in lieu of 208).
509 Bandwidth Limit Exceeded (Apache bw/limited extension)
This status code, while used by many servers, is not specified in any RFCs.
510 Not Extended (RFC 2774)
Further extensions to the request are required for the server to fulfill it.
511 Network Authentication Required (RFC 6585)
The client needs to authenticate to gain network access. Intended for use by intercepting proxies used to control access to the network (e.g. “captive portals” used to require agreement to Terms of Service before granting full Internet access via a Wi-Fi hotspot).
551 Option not supported (RTSP)
An option given in the Require or the Proxy-Require fields was not supported. The Unsupported header should be returned stating the option for which there is no support.
598 Network read timeout error (Unknown)
This status code is not specified in any RFCs, but is used by Microsoft HTTP proxies to signal a network read timeout behind the proxy to a client in front of the proxy.
599 Network connect timeout error (Unknown)
This status code is not specified in any RFCs, but is used by Microsoft HTTP proxies to signal a network connect timeout behind the proxy to a client in front of the proxy.

Social media & sharing icons powered by UltimatelySocial