This is a WIP, but it has actually been implemented and put on production on "tartaglia", a server that hosts several IMCs from italy.

With this guide, you can make your website faster and lighter using varnish as an accelerator and nginx as an ssl proxy

High-level description

That's the setup:

Nginx ---------> Varnish ------> Apache ----> PHP

Well, in theory. But we know that varnish is a beast that must be configured and all the stuff, and we often have sites we do not care/have time to configure with varnish. So, actually, we don't send EVERYTHING to varnish: just what we like to. We decide this basing on the $host variable:

        italy|roma
Nginx ---------------> Varnish ------> Apache ----> PHP
        every other host
Nginx -------------------------------> Apache ----> PHP 

That's because italy and roma are configured to properly use varnish, while the others are not. With this simple rule, we can put sites to production incrementally, one at a time: this makes life easier.

We simplified a bit talking about a simple "italy|roma" rule: there are some other checks that are done, but these are enough for basic understanding. Actually, there are lot of conditions to bypass varnish: logged-in users are recognized by nginx, so varnish is skipped even for italy and roma. Static files are handled by nginx; POSTs skip varnish, too. Note that this will not make it much more efficient (the overhead is minimal), but it will help debugging varnish.

A side note: if you want to put varnish in front of every host, you can put varnish on port 80, making nginx just a "ssl proxy"; this will probably enhance the speed a lot, since varnish is very efficient

Nginx

Nginx just does reverse proxy: this is needed to have ssl (varnish does not support ssl), but it is also good to have a uniform fronted that "dispatches" to various backends. Also, it could serve static files directly (but atm it doesn't).

What should we cache

Basically, everything except pages for logged in users; how to detect logged in users? it depends on the cms you are using: for drupal, you should check the DRUPAL_UID cookie (if using boost); for wordpress, it's the wordpress_logged_in cookie. We discard any other cookie: they're often useless, because we are not interested in tracking users. Also, we do not cache POST or PUT or DELETE. We just cache GET and HEAD. These rules are in varnish, but they are so easy that we should put them in nginx, avoiding some overhead.

Configuration

Nginx

Removing IP

We did not find a direct subsitution for apache mod_remoteip. Anyway, the IP is only put into log: so we just created a new log type called "noip" and use always it. (this actually does more than just removing IP; it also make every request appear as a GET, to hide which browser published posts; this is probably not enough: URIs such as /publish or /node/add should not be logged at all)

log_format  noip '127.0.0.1 $host $remote_user [$time_local] "GET $request_uri" $status $body_bytes_sent "$http_referer" "$http_user_agent" $ssl_cipher $request_time';       

Then you have to specify it.

Also, we need to make sure that the backend does not know the real ip: proxy_set_header X-Real-IP 127.0.0.1

This is not an optimal solution, but is good enough.

SSL

There is a known problem about ssl: not every browser supports SNI, so you can't just have different servers for different servernames: you need to match the host name; it seems simple, but using if is tricky in nginx, so it's not. Let's see how we did it:

server {
    listen 443;                                                                           
        error_page 418 = @varnish;                                                     
                                                                                       
        location / {                                                                   
                                                                                       
                if ( $host ~* "italy.indymedia.org|roma.indymedia.org" ) {             
                        return 418;                                                    
                }                                                                      
                                                                                       
                proxy_pass      http://127.0.0.1:8080/;                                
                proxy_redirect off;                                                    
                proxy_set_header        Host    $host;                                 
                proxy_set_header        X-Real-IP 127.0.0.1;                           
                proxy_set_header        X-Forwarded-Proto https;                       
        }                                                                              
                                                                                       
        location @varnish {                                                            
                proxy_set_header        Host    $host;                                 
                proxy_redirect off;                                                    
                proxy_set_header        X-Real-IP 127.0.0.1;                           
                proxy_set_header        X-Forwarded-Proto https;                       
                                                                                       
                proxy_pass http://127.0.0.1:6081;                                      
                                                                                       
                proxy_connect_timeout   5s;                                            
                proxy_read_timeout      300s; #tantissimo: ci fidiamo di varnish per timeout migliori                                                                         
                                                                                       
        } 
}                                   

The trick is about the return statement. Do you think it sucks?we do, too. There is a simpler solution involving another level of proxying which is planned: we'll document it when we'll test it.

Static files

serving static files from nginx depend on the backend you have; we configured it for both drupal and wordpress. There are lot of guides about doing it.

Varnish

Nothing special: you can copy&paste typical configuration for your cms and you're fine. But debugging varnish is not so easy, so this is a small list of recipes:

Inspect misses

varnishtop -i TxURL -b will show you misses+pass, sorted by frequency. This will start logging by now, so you should wait a few before considering results.

varnishtop -i TxURL -b -1|head -n 20 will show you misses+pass, sorted by frequency, collecting all the data since varnish started.

Munin

Munin is great for inspecting lot of things at a glance: you can compare hit rates, request rates, reason for expunging objects and compare it to most common graph like cpu graph, load average, netstat, etc. This will make it easier to understand what's good and what's wrong with your changes

Inspect bans

Sometimes your cms isn't purging adequately: you can use varnishadm ban.list to check; of course you can grep to search what you need.

Apache

Apache will continue to do its job, but you must move its port to 8080 and remove the ssl server (nginx will do it).

Actually, you can (and should) bind to 127.0.0.1:8080, as it makes no sense to expose a backend outside. Beware, though, that you must then bind every site to 127.0.0.1:8080:if you do this just for a few sites, it won't work as expected.

Backend

Nothing changes here.

Make php recognize ssl

Why?

Sometimes PHP needs to know if the user is hover ssl: for example, it can force some pages on ssl (login, publish page, whatever).

How?

This is usually accomplished using X-Forwarded-Proto: https; anyway, it depends on your cms to recognize it, because the standard php way of understanding it is checking $_SERVER['https']. For wordpress, you can add these lines to the top of your wp-config.php:

if(isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
        $_SERVER['HTTPS'] = 'on';
}

You can probably have it serverside if you are using mod_php, but we are not.

Drupal seems to understand this easily.

-- BoySka - 05 Mar 2012