Caching Youtube Video with Squid and Nginx

Home / Caching Youtube Video with Squid and Nginx

Caching Youtube Video with Squid and Nginx

December 9, 2015 | Article | No Comments

In this article we will discuss about how to cache Youtube Video using Squid as Cache Proxy server and Nginx as Web Server. For this article I use:

  1. Squid
  2. Nginx
  3. Ruby
  4. FreeBSD as server OS

The important part on this article would be Squid, Nginx, and Ruby. If you have installed Squid and Nginx under other Operating System, it’s okay. This method will cover the generic method.

On rest of article, I assume you have installed Squid, Nginx and  Ruby. There is no strict version limitation but I recommend to install the latest version you can find.

Advantages and Disadvantages

In most part of the world, bandwidth is very expensive, especially if you live in developing country. In some scenario, having Youtube or other flash videos cached can speed up loading and save bandwidth. As we might know, a single video might cost a dozen Megabytes or even bigger to download. This method will save the video on cache and if other user request the same video it will be loaded from cache. Working with this method allows us to save much bandwidth, if the content requested is same.

People on same LAN, sometimes watch similar videos. If one put youtube video link on Facebook, Twitter, or other social media, and other people would likely watch that video and that particular video gets viewed many times in few hours. Usually the videos are shared over facebook or other social networking sites so the chances are high for multiple hits per popular videos for my LAN users / friends.

While having an advantage, caching youtube video also has disadvantage.

The chances other user watch the same video is not really high. Or even small. If one search some specific on youtube, more than hundreds result come. The chance another user will search for the same thing and will click on the same link of result is smaller when the result or variation of video is high. That’s why we said the probability is small. For example if we search something and has 300 result, using the most basic probability theorem we get that our probability of clicking the same video is 1/300. Quite small.

If you intend to do youtube caching, you will need fast hardware with tons of space to handle such cache.

Squid Configuration

You can either wipe out your Squid configuration or alter it. Assuming you have configuring something else, better we alter it. This article will gives not specific configuration file but rather pin out what command should we give configuration file.

The default configuration file might be located on /usr/local/etc/squid/squid.conf for FreeBSD. It may be vary if you use other Operating System or installing Squid from source. In the rest of article, we will refer the configuration file as squid.conf.

Now let’s alter the squid.conf, you can adjust the setting:

##
# PORT and Transparent Option
##
http_port 8080 transparent
server_http11 on
icp_port 0

# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log

##
# Cacge object size
##
cache_mem 8 MB
minimum_object_size 0 bytes
maximum_object_size 100 MB
maximum_object_size_in_memory 128 KB

##
# Youtube Cache Section
##
url_rewrite_program /usr/local/etc/nginx/nginx.rb
url_rewrite_host_header off
acl youtube_videos url_regex -i ^http://[^/]+\.youtube\.com/videoplayback\?
acl range_request req_header Range .
acl begin_param url_regex -i [?&]begin=
acl id_param url_regex -i [?&]id=
acl itag_param url_regex -i [?&]itag=
acl sver3_param url_regex -i [?&]sver=3
cache_peer 127.0.0.1 parent 8081 0 proxy-only no-query connect-timeout=10
cache_peer_access 127.0.0.1 allow youtube_videos id_param itag_param sver3_param !begin_param !range_request
cache_peer_access 127.0.0.1 deny all

Now create cache dir and set the permission to proxy.

mkdir /cache1
chown proxy:proxy /cache1
chmod -R 755 /cache1

Then initialize the squid cache directories:

squid -z

Nginx Configuration

Same thing here. You can either wipe out your Nginx configuration or alter it. Assuming you have configuring something else, better we alter it. This article will gives not specific configuration file but rather pin out what command should we give configuration file.

The default configuration file might be located on /usr/local/etc/nginx/nginx.conf for FreeBSD. It may be vary if you use other Operating System or installing Nginx from source. In the rest of article, we will refer the configuration file as nginx.conf.

Now let’s alter the nginx.conf, you can adjust the setting:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
events {
   worker_connections 768;
}
http {
   sendfile on;
   tcp_nopush on;
   tcp_nodelay on;
   keepalive_timeout 65;
   types_hash_max_size 2048;
   include /usr/local/etc/nginx/mime.types;
   default_type application/octet-stream;
   access_log /var/log/nginx/access.log;
   error_log /var/log/nginx/error.log;
   gzip on;
   gzip_static on;
   gzip_comp_level 6;
   gzip_disable .msie6.;
   gzip_vary on;
   gzip_types text/plain text/css text/xml text/javascript application/json application/x-javascript application/xml application/xml+rss;
   gzip_proxied expired no-cache no-store private auth;
   gzip_buffers 16 8k;
   gzip_http_version 1.1;
   include /usr/local/etc/nginx/conf.d/*.conf;
   include /usr/local/etc/nginx/sites-enabled/*;
# starting youtube section
   server {
      listen 127.0.0.1:8081;
      location / {
         root /usr/local/www/nginx_cache/files;
         #try_files "/id=$arg_id.itag=$arg_itag" @proxy_youtube; # Old one
         #try_files  "$uri" "/id=$arg_id.itag=$arg_itag.flv" "/id=$arg_id-range=$arg_range.itag=$arg_itag.flv" @proxy_youtube; #old2
         try_files "/id=$arg_id.itag=$arg_itag.range=$arg_range.algo=$arg_algorithm" @proxy_youtube;
      }
      location @proxy_youtube {
         proxy_pass http://$host$request_uri;
         proxy_temp_path "/usr/local/www/nginx_cache/tmp";
         #proxy_store "/usr/local/www/nginx_cache/files/id=$arg_id.itag=$arg_itag"; # Old 1
         proxy_store "/usr/local/www/nginx_cache/files/id=$arg_id.itag=$arg_itag.range=$arg_range.algo=$arg_algorithm";
         proxy_ignore_client_abort off;
         proxy_method GET;
         proxy_set_header X-YouTube-Cache "[email protected]";
         proxy_set_header Accept "video/*";
         proxy_set_header User-Agent "YouTube Cacher (nginx)";
         proxy_set_header Accept-Encoding "";
         proxy_set_header Accept-Language "";
         proxy_set_header Accept-Charset "";
         proxy_set_header Cache-Control "";
      }
   }
}

Creating File and Folders

In this article we use specific folders for our needs. Better we create and configure it if you don’t have it before.

mkdir /usr/local/www
mkdir /usr/local/www/nginx_cache
mkdir /usr/local/www/nginx_cache/tmp
mkdir /usr/local/www/nginx_cache/files
chown www-data /usr/local/www/nginx_cache/files/ -Rf

Next create a ruby script nginx.rb:

touch /usr/local/etc/nginx/nginx.rb
chmod 755 /usr/local/etc/nginx/nginx.rb

Then edit the file so it would look like this:

#!/usr/bin/env ruby1.8

require "syslog"
require "base64"

class SquidRequest
   attr_accessor :url, :user
   attr_reader :client_ip, :method

   def method=(s)
      @method = s.downcase
   end

   def client_ip=(s)
      @client_ip = s.split('/').first
   end
end

def read_requests
   # URL <SP> client_ip "/" fqdn <SP> user <SP> method [<SP> kvpairs]<NL>
   STDIN.each_line do |ln|
      r = SquidRequest.new
      r.url, r.client_ip, r.user, r.method, *dummy = ln.rstrip.split(' ')
      (STDOUT << "#{yield r}\n").flush
   end
end

def log(msg)
   Syslog.log(Syslog::LOG_ERR, "%s", msg)
end

def main
   Syslog.open('nginx.rb', Syslog::LOG_PID)
   log("Started")

   read_requests do |r|
      if r.method == 'get' && r.url !~ /[?&]begin=/ && r.url =~ %r{\Ahttp://[^/]+\.youtube\.com/(videoplayback\?.*)\z}
         log("YouTube Video [#{r.url}].")
         "http://127.0.0.1:8081/#{$1}"
      else
         r.url
      end
   end
end
main

At this point, we have successfully configuring our server as video caching proxy. You can start the service now.

, ,

About Author

about author

xathrya

A man who is obsessed to low level technology.

Leave a Reply

Your email address will not be published. Required fields are marked *

Social media & sharing icons powered by UltimatelySocial