nginx + unicorn + performance tweaks

nginxThere are several different ways to run your Rails app. Starting from simple  $ rails server and to Phusion Passenger, which is quite complex tools itself. Today, though, I want to focus on nginx.

nginx

In case you’re not familiar with this wonderful tool,

Nginx (pronounced “Engine-X”) is an open source Web server and a reverse proxy server for HTTP, SMTP, POP3 and IMAP protocols, with a strong focus on high concurrency, performance and low memory usage.

The main difference between nginx and Apache is that nginx doesn’t spawn a new process or thread for every new connection. It uses a very tight cycle inside a worker process to handle all connections one-by-one. This way it dramatically decreases the memory footprint required, allowing more concurrent connections.

nginx will be used as a frontend web-server for our Rails webapp.

As you probably know, recently I fell in love with Amazon AWS. It’s all nice, but this kind of relationship comes at a price, and the price tag for running even s1.small instance can be quite high. So here I offer the configuration specifically crafted to efficiently run on t1.micro AWS instance, which is (almost) free to have.

I will explain only relevant configuration lines here, but you can find complete nginx.conf and myapp.conf files at the end of this post.

nginx.conf

Core

First of all, we define a number of worker processes. In case of nginx, the recommendation is to make it equal to the number of processor cores available. As I said before, these workers are not the same as Apache’s, so we can use just one of them:

In order to free some CPU cycles, we will adjust a timer resolution. Basically, it means how often nginx calls  gettimeoftheday() function (no default value, nginx documentation provides 100ms as an example):

We want to be able to open as many files as possible within a single nginx worker process without crashing, so we’re increasing this number (default is 1024, you might also want to increase fs.file-max sysctl param):

To get most of it, we increase the number of simultaneous connections and use  epoll type of polling, which is the ideal method for any Linux 2.6+ distribution (default is 1024 for worker_connections):

Use Host header instead of  server_name for redirects:

Don’t tell the world the intimate details of our nginx installation:

Network

Use Linux  sendfile() for better performance:

We want to send all response headers in one packet. This allows a client to start rendering content immediately after the first packet arrives:

Generally, we want Keep-Alive timeout to be no lower than an average time a user spends on a page before requesting a new one. Google Analytics can give a great idea of what this time is for your website:

Send small data chunks immediately – don’t use Nagle’s Algorithm and increase responsiveness:

As we’re aiming at fast clients, we want to decrease default timeout for waiting client data to save some more memory (default is 60s):

Decrease memory requirements for storing request headers (default is 1024 bytes):

Increase largest allowable request body size. Essentially, this is the maximum size of file upload (default is 1m):

Files

For better performance we enable caching of open file descriptors, information about existence of files/directories, etc. Please note this is not content caching (it uses proxy_cache, which is discussed later):

Cache

In order to increase performance and cut memory footprint we enable content caching. Almost all parameters are quite self-explanatory. levels parameter sets the number of subdirectory levels:

nginx cache uses composite keys to store cached content. We want to slightly improve it to have different cache for different request methods (GET / POST):

And we globally enable proxy caching:

By default, only content with HTTP 200 response codes is cached. The webapp I was developing refreshed information once a day, so it was pretty safe to have all pages cached for 24 hours:

Each time a Rails app tries to set a cookie, even if it’s not used. So we can safely ignore it along with some other headers which prevent nginx from caching. But, please, use this config line with caution, as for many cases it would break your apps!

We want to increase timeout to allow caching engine enough time to answer:

We also slightly increase proxy memory buffer for faster responses:

Compression

We enable compression for network bandwidth optimization and faster response times. Here it’s disabled for broken/unsupported browser and support for HTTP/1.0 is added:

Slightly decrease a number and size of memory buffers used to store compressed data:

Slightly increase compression level (1 is default and the lowest, 9 is the highest):

Compress everything no matter the size (default is 20 bytes):

Compress additional Content Types (default is only text/html):

Enable compression for proxies as well, but do not compress everything. Otherwise we risk confusing remote proxies:

myapp.conf

We use unicorn server as a backend for our Rails webapp with a disabled fail_timeout option:

General server settings:

First we try to open static files, and if that fails we go to the @app section:

Here we have a link to the previously defined upstream server and set necessary HTTP headers to give relevant information to our Rails app:

Here we provide paths for the static content. Please note that we don’t want robots.txt and favicon.ico  requests to pollute our access.log:

And, finally, we gracefully handle any 5xx errors our Rails app might run into:

Before and After — Results

So here is how my global-trend-finder.com webapp did before performance optimizations:

Just terrible, isn’t it?

And what we’ve got after:

Nice difference, eh? Smile

gzip compression allowed to decrease response times three-fold. Another two-fold decrease was from open files caching. But, of course, most of the difference came from proxy_cache.

Files

As promised before, here are the full versions of nginx.conf and myapp.conf files.

7 thoughts on “nginx + unicorn + performance tweaks

  1. Ben

    Hello Vasily,

    thank you very much for these optimisation hints. I finally understood most of the nginx config options thanks to you.

    I was able to increase my requests per second five fold. Impressive.

    If you have the time, would you consider writing a post about how to optimise an nginx + passenger (Ruby Enterprise Edition 1.8.7) production server setup?

    I am running a mixed environment on Ubuntu 12.04. Two websites are Rails 2.3.14 running on nginx + passenger, and one other website is Rails 3.2.9 running on unicorn_rails proxied with nginx.

    I am wondering how the config setting could work together to have all three websites run as fast as possible.

    Cheers
    Ben

    Reply
    1. Vasily Post author

      Hello Ben,

      My pleasure Smile

      As to Passenger, it is somewhat entirely different from unicorn, and it is better suited to run multiple apps from a single host. Anyway, I might consider writing another post covering Passenger + nginx setup very soon.

      In the meantime you can check this guide:
      http://www.modrails.com/documentation/Users%20guide%20Nginx.html

      It’s not focusing on performance, though. But you can use most of this post suggestions, such as enabling proxy_cache and tuning buffer sizes.

      Cheers

      Reply
  2. mikhailov

    it’s great post! Some notices:

    1) “timer_resolution” hasn’t default value, and some people advised to have 100ms at least (obviously 500ms might be better)

    2) have “worker_processes” specific number is not mandatory, since latest Nginx setup it automatically now (http://nginx.org/en/docs/ngx_core_module.html#worker_processes)

    3) “use epoll” setup automatically for Linux machines (http://nginx.org/en/docs/ngx_core_module.html#use)

    4) “worker_rlimit_nofile” doesn’t not big difference until you tune sysctl with “fs.file-max”

    5) “server_tokens off” instead of this you need to change the source before compilation or use “more_set_headers” plugin

    6) “keepalive_timeout” should correspond the minimal time between your users transit between pages, and timeouts don’t save much memory, nginx uses smart algorithm to manage memory. 75s is a pretty standard value.

    7) “open_file_cache” I prefer to use it for static only, since upstream caching is another story

    8) location for static assets can have “gzip_static on” and elimination of Etag generation and Last-Modified filesystem calls. Just think – every time you request a image file, md5 and Last-Modified are requested from the file-system

    I spent more time on optimization Unicorn+Nginx setup, you can find my setup on github if necessary.

    Reply
    1. Vasily Post author

      Hi,

      Thank you for the detailed comments!

      1. You’re right, fixed Smile

      2. Thanks for letting me know. For this type of installation, though, I would keep “1″ instead of auto, because t1.micro AWS instance provides just one (virtual) core, and I’m not sure how nginx autodetect algorithm would work in this case.

      3. I think it’s better to keep “epoll” explicitly specified, as it provides better configuration understanding to anyone who might read it.

      4. In case of t1.micro installation, fs.file-max defaults to “58672″, so we’re OK here Smile I added the comment about this sysctl, though.

      5. Could you please give some more details here? As far as I can say this option does exactly what it says: disables displaying any additional nginx version information on error pages and inside Server HTTP response header.

      6. Added some more details to the post, thank you.

      8. That’s a great tip, thank you! I will try to play around with it to see the difference in response times and system load.

      Cheers,
      Vasily

      Reply
    1. Vasily Post author

      Anatoly – thank you very much for the link and for adding to the conversation! This type of optimizations can be very useful in some environments, indeed.

      I believe, though, that one should be careful with window extending, as it may produce unexpected results over low-quality links – mobile Internet being one example of which.

      Cheers!

      Reply

Leave a Reply