ATT.: Please note that this page is pretty outdated. Check http://sarai.indymedia.org and http://lists.indymedia.org for more current information on this server and contacts.

Sarai Information

Note that what follows far below is for sarai2 which has long been retired. Sarai3 aka sarai has been following as of 10 dec 2004. Since early 2007, sarai3 is out of service and was replaced by sarai4 AKA sarai.

See also: SaraiTips | SaraiMetaInfo

Quick Menu:

Sarai aka Sarai3

Background Information

Sarai is the main IMC mail server, lists.indymedia.org, and has been online since Dec 2004. It replaced sarai2 on Dec 10 Peer1.net (http://www.peer1.net/). Born in late summer of 2004, built from parts of Stallman, Inglis, and other donated items. Placed into rack in downtown seattle sep 2004. Put into service 10 dec 2004.

Sarai is named after a group in India which does fantastic media work, http://www.sarai.net/ .

Operating System

sarai has debian "sarge" installed for most package defaults.
"sarge" was considered "testing" as of dec 2004

mailman is perhaps the only package that was customised.

some hardware info

sarai3 has 2x AMD Athlon(tm) MP 1600 (k7) 1 GB RAM 3ware Storage Controller Driver version: 1.26.00.039 with dual 120 GB disks in raid1 the motherboard and cpu i believe were from inglis.

disk partition, layout, mounts

See also the info about the encrypted partition: SaraiCryptedDisk

Disk /dev/sda: 120.0 GB, 120033075200 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1           4       32098+  83  Linux
/dev/sda2               5         126      979965   83  Linux
/dev/sda3             127         162      289170   83  Linux
/dev/sda4             163       14593   115917007+  8e  Linux LVM

# /etc/fstab: static file system information.
#
#                
proc            /proc           proc    defaults        0       0
/dev/sda2       none            swap    sw              0       0
/dev/sda1       /boot           ext3    defaults        0       2
/dev/sda3       /               ext3    defaults,errors=remount-ro 0       1
/dev/hda        /media/cdrom    iso9660 ro,user,noauto  0       0
#/dev/hda        /cdrom          iso9660 ro,user,noauto  0       0
/dev/fd0        /media/floppy   iso9660 rw,user,noauto  0       0
/dev/vg01/usr   /usr            ext3    rw                      0       2
/dev/vg01/home  /home           ext3    rw                      0       2
/dev/vg01/mailman /mailman      ext3    rw,nodev,sync,noatime   0       2
/dev/vg01/var   /var            ext3    rw                      0       2

# mount
/dev/sda3 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext3 (rw)
/dev/mapper/vg01-usr on /usr type ext3 (rw)
/dev/mapper/vg01-home on /home type ext3 (rw)
/dev/mapper/vg01-mailman on /mailman type ext3 (rw,nodev,sync,noatime)
/dev/mapper/vg01-var on /var type ext3 (rw)

#df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda3                265M  104M  147M  42% /
tmpfs                    507M     0  507M   0% /dev/shm
/dev/sda1                30M   14M   14M   50% /boot
/dev/mapper/vg01-usr     5.8G  787M  4.8G  15% /usr
/dev/mapper/vg01-home    2.0G  659M  1.2G  36% /home
/dev/mapper/vg01-mailman 68G   34G   33G   51% /mailman
/dev/mapper/vg01-var     2.9G  501M  2.3G  18% /var

grub information

two main kernels installed originally:

title           Debian GNU/Linux, kernel 2.6.8-1-k7
root            (hd0,0)
kernel          /vmlinuz-2.6.8-1-k7 root=/dev/sda3 ro
initrd          /initrd.img-2.6.8-1-k7
savedefault
boot

title           Debian GNU/Linux, kernel 2.6.3-1-386
root            (hd0,0)
kernel          /vmlinuz-2.6.3-1-386 root=/dev/sda3 ro
initrd          /initrd.img-2.6.3-1-386
savedefault
boot

kernel config for 2.6.8-1-k7 (from deb kernel image package):

• config-2.6.8-1-k7.gz: kernel config for 2.6.8-1-k7 from deb kernel image package

Network and environmental stats

Mailman Queue Statistics are here: http://sarai.indymedia.org/cgi-bin/mailmanqueues.cgi

Postfix Queue Statistics are here (not working at the moment 23/11/06): http://sarai.indymedia.org/cgi-bin/queuegraph.cgi

Mail Statistics are here: http://sarai.indymedia.org/cgi-bin/mailgraph.cgi

Munin Sarai3: http://volunteer.indymedia.org/munin/indymedia.org/sarai3.indymedia.org.html

Munin sarai (old): http://volunteer.indymedia.org/munin/indymedia.org/sarai.indymedia.org.html

-- StefaniB - 12 Dec 2004

Sarai2

Background Information

Sarai2 was the main IMC mail server, lists.indymedia.org, and was online from January 2002 until Dec 2004. It replaced sarai1 on May 23rd 2002 and was hosted at Galaxy-7/Cortland. Though we had a number of problems with it, the hard work of deanna, stefani, micah, mtoups and others made sarai a much more reliable addition to the network.

basic info

The new sarai is an HP netserver LH3, located at http://www.galaxy-7.net, its a big machine, weighing in the far corner at: 35.5 cm wide, 48.2 cm tall, 73.6 cm deep (14" wide, 19" tall, 29" deep)

firmware version D.03.02  
bios ver B.02.01. sep 2 1999
Version b.02.02 sep 01 1999 netraid express
phoenix bios 4.06.36 PL
netserver management controller firmware: b.05.08

Netserver RAID Setup

during the POST, <control> -M gets you into the Netserver Utility.

it will present a menu

select Objects--> Physical Drives  
to see the status of the disks.

if they say "fail" you can use the arrow keys to select the item 
you want, and hit the <enter> key.  there is an option 
"make online" which , if the drive is really ok,  makes the 
disk available.

Netserver RAID Discussion

http://forums1.itrc.hp.com/service/forums/questionanswer.do?admit=716493758+1065719678972+28353475&threadId=229164

Sarai2 Tips

1. If load is spiking, check to see if a robot is driving up the load by requesting Mailman CGIs

2. When re-starting apache use /var/www/bin/apachectl startssl

Sarai2 Kernel Recompile

A special kernel with raid support for the hardware raid card had to be compiled, and certain raid optimizations had to be turned off to get the disk performance to work properly. See NewSaraiKernelConfig

SaraiMetaInfo

Sarai Backup Info

List archives, data and configs are being backed up to emma twice daily at 5am and 5pm PST via rsync

Miscellaneous

Contact

Sarai is maintained by both the listwork (listwork@lists.indymedia.org) working group. Before reporting a potential problem, check out ServerStatus to see if the problem is already known.

When sarai is down, contact imc-tech-emerg@lists.cat.org.au.

Listwork Howto's and Errata are available at ListworkWorkingGroup

Common Problems and Uncommon Solutions

SpamDOutofControl - a previously common problem where spamd would take more than its fair share of CPU cycles

Outstanding Issues

Old sarai background (defunct)

The old Sarai was a black rackmount case, a 2U size, that was apparantly located in rack 03-024. At Datarealm: http://www.datarealm.com/

Due to the nature of the case, the utmost care should be taken when attaching a monitor cable to the video card. The video card can become unseated very easily. The top of the case was also suspect, if it were pushed down it could cause some issues. The entire case was a nightmare to assemble; the motherboard did not align with the case, for example, so the case had to be modified with metalworking to make it fit. In the end everything didn't fit quite right, a riser card was necessary to get the video card and NIC to work; the top of the case appeared bent. If things were put on top of the case, that might have caused the problem as well.

Sarai's inception was a troubled one. When the machine was first built, we had many problems getting the case to work right, we ended up with a case that is not in the best of shape. When it was first brought online, after it started getting some traffic, it would crash almost every other day. This required shoji to physically reboot the machine. This was not good. Micah recompiled the kernel to have software watchdog support built-in (http://www.linuxhq.com/kernel/v2.4/doc/watchdog.txt.html). This provides a device, /dev/watchdog, which when open must be written to within a timeout or the machine will reboot. Each write delays the reboot time another timeout. In the case of the software watchdog the ability to reboot will depend on the state of the machines and interrupts. The hardware boards physically pull the machine down off their own onboard timers and will reboot from almost anything, we don't have a hardware board, but the software one did the trick. No more physical intervention required by shoji, the machine would reboot itself as needed. While this was going on Micah frantically searched for a cause and solution to this problem.

After many troubleshooting steps, Micah decided that it could be a hardware problem. He has seen this sort of behavior when there is bad memory involved. He got some new memory and ran http://www.teresaudio.com/memtest86/ on it for 24 hours. Whenever you build a server it is HIGHLY recommended that you test the memory before you deploy it! This was a learning experience. inglis had her memory tested before we put it into place, this is good practice. So he found some memory that passed the test and sent it out to replace the existing memory. This worked! Sarai no longer crashed.

It has been running fine for about 4 months now, but something has started to happen again to sarai. We had a problem where the power cord was bumped out, this was easily done because it didn't seem to seat securely, I thought that this was resolved, but it may not have been.

Thinking of putting http://lwn.net/2001/0412/a/oom-patch.php3 on to the machine. We are stuck now because the remote machine seems to be OOM and I can't ssh in (sits there on debug: Connection established. and never continues), if I could set aside some memory for ssh, login, su, ps and kill, then I could remotely fix this problem.

-- MatRiseup - 21 Jun 2004 - Added statistics links