Please note that this page is pretty outdated. Check http://sarai.indymedia.org
for more current information on this server and contacts.
Note that what follows far below is for sarai2 which has long been retired. Sarai3 aka sarai has been following as of 10 dec 2004. Since early 2007, sarai3 is out of service and was replaced by sarai4 AKA sarai.
See also: SaraiTips
Sarai aka Sarai3
Sarai is the main IMC mail server, lists.indymedia.org, and has been online since Dec 2004. It replaced sarai2 on Dec 10 Peer1.net (http://www.peer1.net/
). Born in late summer of 2004, built from parts of Stallman, Inglis, and other donated items. Placed into rack in downtown seattle sep 2004. Put into service 10 dec 2004.
Sarai is named after a group in India which does fantastic media work, http://www.sarai.net/
sarai has debian "sarge" installed for most package defaults.
"sarge" was considered "testing" as of dec 2004
mailman is perhaps the only package that was customised.
some hardware info
2x AMD Athlon(tm) MP 1600 (k7)
1 GB RAM
3ware Storage Controller Driver version: 1.26.00.039 with dual 120 GB disks in raid1
the motherboard and cpu i believe were from inglis.
disk partition, layout, mounts
See also the info about the encrypted partition: SaraiCryptedDisk
Disk /dev/sda: 120.0 GB, 120033075200 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 4 32098+ 83 Linux
/dev/sda2 5 126 979965 83 Linux
/dev/sda3 127 162 289170 83 Linux
/dev/sda4 163 14593 115917007+ 8e Linux LVM
# /etc/fstab: static file system information.
proc /proc proc defaults 0 0
/dev/sda2 none swap sw 0 0
/dev/sda1 /boot ext3 defaults 0 2
/dev/sda3 / ext3 defaults,errors=remount-ro 0 1
/dev/hda /media/cdrom iso9660 ro,user,noauto 0 0
#/dev/hda /cdrom iso9660 ro,user,noauto 0 0
/dev/fd0 /media/floppy iso9660 rw,user,noauto 0 0
/dev/vg01/usr /usr ext3 rw 0 2
/dev/vg01/home /home ext3 rw 0 2
/dev/vg01/mailman /mailman ext3 rw,nodev,sync,noatime 0 2
/dev/vg01/var /var ext3 rw 0 2
/dev/sda3 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext3 (rw)
/dev/mapper/vg01-usr on /usr type ext3 (rw)
/dev/mapper/vg01-home on /home type ext3 (rw)
/dev/mapper/vg01-mailman on /mailman type ext3 (rw,nodev,sync,noatime)
/dev/mapper/vg01-var on /var type ext3 (rw)
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 265M 104M 147M 42% /
tmpfs 507M 0 507M 0% /dev/shm
/dev/sda1 30M 14M 14M 50% /boot
/dev/mapper/vg01-usr 5.8G 787M 4.8G 15% /usr
/dev/mapper/vg01-home 2.0G 659M 1.2G 36% /home
/dev/mapper/vg01-mailman 68G 34G 33G 51% /mailman
/dev/mapper/vg01-var 2.9G 501M 2.3G 18% /var
two main kernels installed originally:
title Debian GNU/Linux, kernel 2.6.8-1-k7
kernel /vmlinuz-2.6.8-1-k7 root=/dev/sda3 ro
title Debian GNU/Linux, kernel 2.6.3-1-386
kernel /vmlinuz-2.6.3-1-386 root=/dev/sda3 ro
kernel config for 2.6.8-1-k7 (from deb kernel image package):
Network and environmental stats
Mailman Queue Statistics are here: http://sarai.indymedia.org/cgi-bin/mailmanqueues.cgi
Postfix Queue Statistics are here (not working at the moment 23/11/06)
Mail Statistics are here:
Munin sarai (old):
- 12 Dec 2004
Sarai2 was the main IMC mail server, lists.indymedia.org, and was online from January 2002 until Dec 2004. It replaced sarai1 on May 23rd 2002 and was hosted at Galaxy-7/Cortland. Though we had a number of problems with it, the hard work of deanna, stefani, micah, mtoups and others made sarai a much more reliable addition to the network.
The new sarai is an HP netserver LH3, located at http://www.galaxy-7.net
, its a big machine, weighing in the far corner at: 35.5 cm wide, 48.2 cm tall, 73.6 cm deep (14" wide, 19" tall, 29" deep)
firmware version D.03.02
bios ver B.02.01. sep 2 1999
Version b.02.02 sep 01 1999 netraid express
phoenix bios 4.06.36 PL
netserver management controller firmware: b.05.08
Netserver RAID Setup
during the POST, <control> -M gets you into the Netserver Utility.
it will present a menu
select Objects--> Physical Drives
to see the status of the disks.
if they say "fail" you can use the arrow keys to select the item
you want, and hit the <enter> key. there is an option
"make online" which , if the drive is really ok, makes the
Netserver RAID Discussion
1. If load is spiking, check to see if a robot is driving up the load by requesting Mailman CGIs
2. When re-starting apache use /var/www/bin/apachectl startssl
Sarai2 Kernel Recompile
A special kernel with raid support for the hardware raid card had to be compiled, and certain raid optimizations had to be turned off to get the disk performance to work properly. See NewSaraiKernelConfig
Sarai Backup Info
List archives, data and configs are being backed up to emma twice daily at 5am and 5pm PST via rsync
Sarai is maintained by both the listwork (firstname.lastname@example.org
) working group. Before reporting a potential problem, check out ServerStatus
to see if the problem is already known.
When sarai is down, contact email@example.com
Listwork Howto's and Errata are available at ListworkWorkingGroup
Common Problems and Uncommon Solutions
- a previously common problem where spamd would take more than its fair share of CPU cycles
Old sarai background (defunct)
The old Sarai was a black rackmount case, a 2U size, that was apparantly located in rack 03-024. At Datarealm: http://www.datarealm.com/
Due to the nature of the case, the utmost care should be taken when attaching a monitor cable to the video card. The video card can become unseated very easily. The top of the case was also suspect, if it were pushed down it could cause some issues. The entire case was a nightmare to assemble; the motherboard did not align with the case, for example, so the case had to be modified with metalworking to make it fit. In the end everything didn't fit quite right, a riser card was necessary to get the video card and NIC to work; the top of the case appeared bent. If things were put on top of the case, that might have caused the problem as well.
Sarai's inception was a troubled one. When the machine was first built, we had many problems getting the case to work right, we ended up with a case that is not in the best of shape. When it was first brought online, after it started getting some traffic, it would crash almost every other day. This required shoji to physically reboot the machine. This was not good. Micah recompiled the kernel to have software watchdog support built-in (http://www.linuxhq.com/kernel/v2.4/doc/watchdog.txt.html
). This provides a device, /dev/watchdog, which when open must be written to within a timeout or the machine will reboot. Each write delays the reboot time another timeout. In the case of the software watchdog the ability to reboot will depend on the state of the machines and interrupts. The hardware boards physically pull the machine down off their own onboard timers and will reboot from almost anything, we don't have a hardware board, but the software one did the trick. No more physical intervention required by shoji, the machine would reboot itself as needed. While this was going on Micah frantically searched for a cause and solution to this problem.
After many troubleshooting steps, Micah decided that it could be a hardware problem. He has seen this sort of behavior when there is bad memory involved. He got some new memory and ran http://www.teresaudio.com/memtest86/
on it for 24 hours. Whenever you build a server it is HIGHLY
recommended that you test the memory before you deploy it! This was a learning experience. inglis had her memory tested before we put it into place, this is good practice. So he found some memory that passed the test and sent it out to replace the existing memory. This worked! Sarai no longer crashed.
It has been running fine for about 4 months now, but something has started to happen again to sarai. We had a problem where the power cord was bumped out, this was easily done because it didn't seem to seat securely, I thought that this was resolved, but it may not have been.
Thinking of putting http://lwn.net/2001/0412/a/oom-patch.php3
on to the machine. We are stuck now because the remote machine seems to be OOM and I can't ssh in (sits there on debug: Connection established. and never continues), if I could set aside some memory for ssh, login, su, ps and kill, then I could remotely fix this problem.
- 21 Jun 2004 - Added statistics links