Kompost : IMC-docs Server

Kompost is docs.indymedia.org's dedicated server, hosted in the UK.

Monitoring

See ImcDocs#KompostInfo_kompost.

Collective sysadmin

Filesystem organization

• host system Changelogs & al. : kompost:/root/admin/
• host system SERVICE Changelog : kompost:/root/admin/SERVICE/Changelog
• BLABLA vserver Changelogs & al. : BLABLA:/root/admin/BLABLA

Tools

We're using metche, which is a system that monitors *nix servers configuration file for easy change tracking and team coordination. Its most helpful feature is the automatic sending of change reports.

There is a conf file in /etc/metche/ per system to monitor ; everything is run from /etc/cron.d/metche.

Email

As root, we receive tons of emails from various sources ; so that they're managed collectively, and so that each admin only receive the emails about the vservers he/she actually admins, there is a root-HOSTNAME alias for each vserver, and a global root alias for the people admin'ing the host system : kompost:/etc/aliases.

Hardware

• CHASSIS: 0.5U / Aluminium Thermal Chassis / CD
• CPU: P4/2.8GHz
• RAM: 1Gb
• HDD: 2 x 120Gb IDE
• MOTHERBOARD: Gigabyte GA-8I845GV

Software

Base system

OS

Debian Sarge on Software RAID-1.

Network

• default route : ?
• nameservers : 4.2.2.1
• IPs : use ?

Kernel

How to upgrade kompost kernel ? Explain in /root/admin/common/Changelog what you have installed.

Partitions

• / : 500MB - ext3
• /boot : 100MB - ext3
• /home : 20GB - ext3
• /usr : 5GB - ext3
• /tmp : 200MB - ext3
• swap : 1GB
• /var : 5GB, using lvm
• /var/lib/vservers/humus : 25GB, encrypted lvm
• /var/lib/vservers/fungi : 10GB, encrypted lvm
• /var/lib/vservers/lombric : 5GB, encrypted lvm

the encrypted partitions will only be able to be mounted after the machine is up and running. to start them, log in, become root, and do this:

[ /etc/init.d/cryptdisks start

it will prompt you for a passphrase - you have a limited time to enter it, so be warned!! following this, the partition can be mounted and the vserver started

Apache / Perl

• I guess the config files currently in use are ok. -- IntRigeri - 12 Apr 2005

TWiki

The currently running setup is described on HumusInfo.

Mail setup

The host system (kompost) runs Postfix with TLS support.

All vservers run nullmailer, a simple relay-only mail transport agent ; they use kompost as smarthost. Their (shared) config is nullmailer-vserver-config.tgz.

Vservers

Our setup

The system hosts the following vservers :

• humus : dedicated to docs.indymedia.org hosting / 89.107.22.13
• fungi : dedicated to dev.docs.indymedia.org hosting / 89.107.22.14
• lombric : was dedicated to run a rsync server for backups / 89.107.22.15 / but has now been handed over to the IrcDWorkingGroup for use as a european node.

Some useful doc

• linux-vserver patch & utils homepage
• riseup doc
• Gentoo VServer Handbook

Monitoring the vservers : Intrusion detection system (IDS)

Running

• rkhunter (rootkit hunter)
• chkrootkit (rootkit hunter)
• logcheck (log checker) ; don't forget to :
  • add to /etc/logcheck/logcheck.logfiles the vservers logs (such as /var/log/vservers/10.0.0.4/syslog)
  • add to the logcheck rules the ones suitable for the daemons running inside the vservers
• tiger (Unix security checker)

Half-installed

• integrit (files integrity checker) ; please read KompostIntegritHowTo to know how to work with this necessary crap

To look at

Network IDS

• snort

Misc tools

• idswakeup : NIDS tester
• mod_evasive: a Debian package is in sid
• mod_security: would be cool to use this
• bastille
• john

No

• portsentry : no need to monitor portscans when nearly all our ports are closed at the firewall level
• prelude : the Debian maintainer has not upgraded the packages for ages

Logging

vserver side

The vservers send their logs to the host box.

Commented all lines of /etc/syslog.conf but :

# replace the following IP with the host box IP
*.*         @193.123.432.123
*.emerg     *

host system side

The host box filters, sorts and stores all these logs it receives, thanks to syslog-ng, that is able to sort in function of source IP.

See also riseup's config.

Some relevant syslog-ng config parts :

destination df_vservers {                                                                                                              file("/var/log/vservers/$HOST/$FACILITY.log" owner(root) group(adm) perm(0640) dir_perm(0755) create_dirs(yes)); 
};    
destination df_vservers_syslog {                                                                                                              file("/var/log/vservers/$HOST/syslog" owner(root) group(adm) perm(0640) dir_perm(0755) create_dirs(yes)); 
};    
destination df_vservers_www {                                                                                                          file("/var/log/vservers/$HOST/apache/$PROGRAM/access.log" owner(root) group(adm) perm(0640) dir_perm(0755) create_dirs(yes));
};

filter f_greenvserver {                                                                                                                             host(10.0.0.4); 
};
filter f_bluevserver {                                                                                                                             host(10.0.0.5); 
};
filter f_vservers {
        filter(f_bluevserver)
        or filter(f_greenvserver)
        and not filter(f_local);
};
filter f_vservers_www {
        filter(f_vservers)
        and program("mail.greenvserver.taz")
        or program("www.bluevserver.taz");
};

log {
        source(s_all);
        filter(f_vservers_www);
        destination(df_vservers_www);
        flags(final);
};
log {   
        source(s_all);
        filter(f_vservers);
        filter(f_syslog);
        destination(df_vservers_syslog);
};
log {
        source(s_all);
        filter(f_vservers);
        destination(df_vservers);
        flags(final);
};

This way, we get on the host filesystem a tree like this :

ls -lR /var/log/vservers/10.0.0.4
/var/log/vservers/10.0.0.4:
total 16K
drwxr-xr-x  3 root root 4,0K 2005-04-13 02:59 apache
-rw-r-----  1 root adm   131 2005-04-13 13:36 authpriv.log
-rw-r-----  1 root adm  1,3K 2005-04-13 13:22 kern.log
-rw-r-----  1 root adm   212 2005-04-13 13:36 syslog
-rw-r-----  1 root adm   212 2005-04-13 13:36 syslog.log

/var/log/vservers/10.0.0.4/apache:
total 4,0K
drwxr-xr-x  2 root root 4,0K 2005-04-13 13:43 mail.greenvserver

/var/log/vservers/10.0.0.4/apache/mail.greenvserver:
total 4,0K
-rw-r-----  1 root adm 393 2005-04-13 13:43 access.log

The Apache problem

By default, Apache does not use syslog ; solution : pipe its logs to a script that will do this :

### log to the main server
LogFormat "127.0.0.1 %v \"%r\" %>s %b \"%{Referer}i\"" privacy_format
CustomLog "| /usr/bin/logger -t docs.indymedia.org -p local1.info" privacy_format
LogLevel error
ErrorLog "| /usr/bin/logger -t docs.indymedia.org -p local2.info"

Problem : by default, syslog-ng does not use the Apache log format because it adds some additional info at the beggining of each line. Might be a problem for statistics programs ; anyway, Awstats allows us to tell him exactly which format is used.

rotating logs

cf http://deb.riseup.net/logging/rotating/

awstats

The abandonned web stats setup is documented on KompostOldStatsSetup.

old stuff

Some Notes

With Debian Etch having entered 'stable' state, it is very probably no longer neccessary to compile the kernel ourselves, but stock kernel vserver images can be used. This would allow for easier upgrading in case of security vulnerabilities are discovered, as well as to get aware of and track them in the first place.

The previously neccessary requirements, such as the Bind Mount Extensions (BME) have been integrated into the linux-vserver patches 2.0.2 and higher, which are used in the stock linux-image-2.6-vserver images. On the other hand, BME is no longer a requirement for kompost anyway, as BME is no longer used.

Note that this needs yet to be tested to be really sure at this point.

Upgrade util-vserver

The VServer userspace utilities have to be kept in sync with the VServer kernel patch we use, that's why the first thing to do is sudo apt-get update ; sudo apt-get install util-vserver, which will upgrade this package to the latest backports.org's one.

Get the source, Luke

We need to find out what's the current best combination of kernel version and VServer patch. This process is more or less voodoo.

Information and tarball providers :

• http://kernel.org to know the latest upstream kernel version
• http://linux-vserver.org/ to know the latest VServer patches versions (we're using the 2.0 branch, which includes the Bind Mount Extensions we need)
• http://packages.qa.debian.org/linux-2.6 to know which kernel version is available in sid ; then sudo apt-get install linux-source-2.6.XX, replacing XX by the latest number available
• sudo apt-get install kernel-patch-vserver to upgrade the Debian package to the newest sid's one, then have a look into /usr/src/kernel-patches/diffs/vserver to know which patch version is included in this package

A few tips :

• We prefer to use the Debian kernel sources if feasible, but we sometimes have to use the upstream ones, for example if a security fix is not available yet in Debian.
• The next steps are easier (and more elegant, too) when using the Debian packaged kernel-patch-vserver, but we sometimes prefer to use a newer upstream patch, for whatever reason.

Explain in /root/admin/common/Changelog which kernel and patch versions you've chosen and downloaded.

Sources extraction, patching and configuration

1. Extract the chosen and downloaded Linux source tarball, in /usr/src.
2. If the chosen VServer patch is the upstream one, and not the Debian packaged one, apply it to this brand new source tree.
3. Copy the running kernel's configuration file (e.g. /boot/config-2.6.16.10-vs2.0.2-rc18) to the root of this source tree, naming it .config
4. In this kernel source tree, run make oldconfig in order to update the kernel config

Compilation

We usually name our kernels with the upstream source version and the VServer patch version ; e.g. kernel-image-2.6.16.10-vs2.0.2-rc18. Usually, the VServer patch modifies the kernel Makefile in order to add its version (e.g. -vs2.0.2-rc18) to the kernel name, so this should not require any manual intervention ; in case it does not work as flawlessly as expected, add the --append-to-version -vs2.0.2-rc18 option to make-kpkg command line (replacing -vs2.0.2-rc18 by the VServer patch version, of course).

To ease kernel maintenance, we want to build and install our kernels as Debian packages. The make-kpkg command builds a Debian package, when run at the root directory of a kernel source tree.

If the chosen VServer patch is the upstream one, it's supposed to have been already applied if you're following this howto. The command to run is thus : make-kpkg --rootcmd fakeroot binary

Else, the command to run is : make-kpkg --rootcmd fakeroot --added-patches vserver binary

Note : until 2.6.13, we used to add --initrd to this command line, so that an initrd is built. I (intrigeri) forgot it for 2.6.16, and it seems to work alright without it ; so I guess we actually don't need it anymore, we just have to make sure every driver needed on first boot stages (such as the root FS and IDE controller ones...) is compiled inside the kernel, and not as a module.

New kernel setup

After make-kpkg has finished its job, /usr/src/ contains a few brand new Debian packages : kernel-doc-VERSION.deb, kernel-headers-VERSION.deb, kernel-image-VERSION.deb and kernel-source-VERSION.deb. Install (with dpkg -i) the kernel-image and kernel-source one ; the kernel-image contains the previously compiled kernel and modules, the kernel-source contains the source used for compilation, with all used patches applies ; it's great to keep this one along with a kernel since you sometimes need your currently running kernel's sources to build another piece of software.

The kernel-image installation with dpkg has automatically added two entries at the top of /boot/grub/menu.lst ; these two entries (wich we'll call number 0 and 1) are :

• 0 : the newly installed kernel, in normal mode
• 1 : the newly installed kernel, in single mode (rescue mode)

And thus the current kernel is now the third and fourth entries (numbers 2 and 3).

We're going to setup grub so that the box will reboot once on the new kernel (entry 0), and then on the old (and known working) one (entry 2) ; this way, if the new kernel does not boot, somebody will go to the colo and reboot kompost, which will nicely reboot on a know working kernel) :

• edit /boot/grub/menu.lst to change the "default 0" line to "default 2", so that the known working kernel is the default one
• run sudo grub-reboot 0 so that the new kernel (entry 0) is used on next reboot only

The box can now be rebooted on the new kernel :

1. Drop a line to #tech and #twiki so that nobody is surprised by the box being down for a little while.
2. Stop the running VServers first (with sudo vserver NAME stop), since I'm not sure our VServer config does so automatically.
3. sudo reboot
4. Wait a few dozens seconds, then login via ssh
5. Save the kernel boot messages : dmesg > /root/admin/common/dmesg/dmesg-YYYYMMDD
6. Starts the VServers if they're not running yet.
7. Have a look to syslog.
8. If everything seems to be running smoothly, set the new kernel (entry 0) as the default Grub one in /boot/grub/menu.lst ; else, reboot the box on the old and known working kernel.
9. Finish reporting to /root/admin/common/Changelog what you've done.

• /var/lib/vsersers : 20GB - ext3
• /var : what's left - ext3