ATT.: Please note that this page is pretty outdated. Check
http://sarai.indymedia.org and
http://lists.indymedia.org for more current information on this server and contacts.
Note that what follows far below is for sarai2 which has long been retired. Sarai3 aka sarai has been following as of 10 dec 2004. Since early 2007, sarai3 is out of service and was replaced by sarai4 AKA sarai.
See also:
SaraiTips |
SaraiMetaInfo
Quick Menu:
Sarai aka Sarai3
Sarai is the main IMC mail server, lists.indymedia.org, and has been online since Dec 2004. It replaced sarai2 on Dec 10 Peer1.net (
http://www.peer1.net/). Born in late summer of 2004, built from parts of Stallman, Inglis, and other donated items. Placed into rack in downtown seattle sep 2004. Put into service 10 dec 2004.
Sarai is named after a group in India which does fantastic media work,
http://www.sarai.net/ .
Operating System
sarai has debian "sarge" installed for most package defaults.
"sarge" was considered "testing" as of dec 2004
mailman is perhaps the only package that was customised.
some hardware info
sarai3 has
2x AMD Athlon(tm) MP 1600 (k7)
1 GB RAM
3ware Storage Controller Driver version: 1.26.00.039 with dual 120 GB disks in raid1
the motherboard and cpu i believe were from inglis.
disk partition, layout, mounts
See also the info about the encrypted partition:
SaraiCryptedDisk
Disk /dev/sda: 120.0 GB, 120033075200 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 4 32098+ 83 Linux
/dev/sda2 5 126 979965 83 Linux
/dev/sda3 127 162 289170 83 Linux
/dev/sda4 163 14593 115917007+ 8e Linux LVM
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/sda2 none swap sw 0 0
/dev/sda1 /boot ext3 defaults 0 2
/dev/sda3 / ext3 defaults,errors=remount-ro 0 1
/dev/hda /media/cdrom iso9660 ro,user,noauto 0 0
#/dev/hda /cdrom iso9660 ro,user,noauto 0 0
/dev/fd0 /media/floppy iso9660 rw,user,noauto 0 0
/dev/vg01/usr /usr ext3 rw 0 2
/dev/vg01/home /home ext3 rw 0 2
/dev/vg01/mailman /mailman ext3 rw,nodev,sync,noatime 0 2
/dev/vg01/var /var ext3 rw 0 2
# mount
/dev/sda3 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext3 (rw)
/dev/mapper/vg01-usr on /usr type ext3 (rw)
/dev/mapper/vg01-home on /home type ext3 (rw)
/dev/mapper/vg01-mailman on /mailman type ext3 (rw,nodev,sync,noatime)
/dev/mapper/vg01-var on /var type ext3 (rw)
#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 265M 104M 147M 42% /
tmpfs 507M 0 507M 0% /dev/shm
/dev/sda1 30M 14M 14M 50% /boot
/dev/mapper/vg01-usr 5.8G 787M 4.8G 15% /usr
/dev/mapper/vg01-home 2.0G 659M 1.2G 36% /home
/dev/mapper/vg01-mailman 68G 34G 33G 51% /mailman
/dev/mapper/vg01-var 2.9G 501M 2.3G 18% /var
two main kernels installed originally:
title Debian GNU/Linux, kernel 2.6.8-1-k7
root (hd0,0)
kernel /vmlinuz-2.6.8-1-k7 root=/dev/sda3 ro
initrd /initrd.img-2.6.8-1-k7
savedefault
boot
title Debian GNU/Linux, kernel 2.6.3-1-386
root (hd0,0)
kernel /vmlinuz-2.6.3-1-386 root=/dev/sda3 ro
initrd /initrd.img-2.6.3-1-386
savedefault
boot
kernel config for 2.6.8-1-k7 (from deb kernel image package):
Network and environmental stats
Mailman Queue Statistics are here:
http://sarai.indymedia.org/cgi-bin/mailmanqueues.cgi
Postfix Queue Statistics are here
(not working at the moment 23/11/06):
http://sarai.indymedia.org/cgi-bin/queuegraph.cgi
Mail Statistics are here:
http://sarai.indymedia.org/cgi-bin/mailgraph.cgi
Munin Sarai3:
http://volunteer.indymedia.org/munin/indymedia.org/sarai3.indymedia.org.html
Munin sarai (old):
http://volunteer.indymedia.org/munin/indymedia.org/sarai.indymedia.org.html
--
StefaniB - 12 Dec 2004
Sarai2
Sarai2 was the main IMC mail server, lists.indymedia.org, and was online from January 2002 until Dec 2004. It replaced sarai1 on May 23rd 2002 and was hosted at Galaxy-7/Cortland. Though we had a number of problems with it, the hard work of deanna, stefani, micah, mtoups and others made sarai a much more reliable addition to the network.
basic info
The new sarai is an HP netserver LH3, located at
http://www.galaxy-7.net, its a big machine, weighing in the far corner at: 35.5 cm wide, 48.2 cm tall, 73.6 cm deep (14" wide, 19" tall, 29" deep)
firmware version D.03.02
bios ver B.02.01. sep 2 1999
Version b.02.02 sep 01 1999 netraid express
phoenix bios 4.06.36 PL
netserver management controller firmware: b.05.08
Netserver RAID Setup
during the POST, <control> -M gets you into the Netserver Utility.
it will present a menu
select Objects--> Physical Drives
to see the status of the disks.
if they say "fail" you can use the arrow keys to select the item
you want, and hit the <enter> key. there is an option
"make online" which , if the drive is really ok, makes the
disk available.
Netserver RAID Discussion
http://forums1.itrc.hp.com/service/forums/questionanswer.do?admit=716493758+1065719678972+28353475&threadId=229164
Sarai2 Tips
1. If load is spiking, check to see if a robot is driving up the load by requesting Mailman CGIs
2. When re-starting apache use /var/www/bin/apachectl startssl
Sarai2 Kernel Recompile
A special kernel with raid support for the hardware raid card had to be compiled, and certain raid optimizations had to be turned off to get the disk performance to work properly. See
NewSaraiKernelConfig
Sarai Backup Info
List archives, data and configs are being backed up to emma twice daily at 5am and 5pm PST via rsync
Miscellaneous
Sarai is maintained by both the listwork (
listwork@lists.indymedia.org) working group. Before reporting a potential problem, check out
ServerStatus to see if the problem is already known.
When sarai is down, contact
imc-tech-emerg@lists.cat.org.au.
Listwork Howto's and Errata are available at
ListworkWorkingGroup
Common Problems and Uncommon Solutions
SpamDOutofControl - a previously common problem where spamd would take more than its fair share of CPU cycles
Outstanding Issues
Old sarai background (defunct)
The old Sarai was a black rackmount case, a 2U size, that was apparantly located in rack 03-024. At Datarealm:
http://www.datarealm.com/
Due to the nature of the case, the utmost care should be taken when attaching a monitor cable to the video card. The video card can become unseated very easily. The top of the case was also suspect, if it were pushed down it could cause some issues. The entire case was a nightmare to assemble; the motherboard did not align with the case, for example, so the case had to be modified with metalworking to make it fit. In the end everything didn't fit quite right, a riser card was necessary to get the video card and NIC to work; the top of the case appeared bent. If things were put on top of the case, that might have caused the problem as well.
Sarai's inception was a troubled one. When the machine was first built, we had many problems getting the case to work right, we ended up with a case that is not in the best of shape. When it was first brought online, after it started getting some traffic, it would crash almost every other day. This required shoji to physically reboot the machine. This was not good. Micah recompiled the kernel to have software watchdog support built-in (
http://www.linuxhq.com/kernel/v2.4/doc/watchdog.txt.html). This provides a device, /dev/watchdog, which when open must be written to within a timeout or the machine will reboot. Each write delays the reboot time another timeout. In the case of the software watchdog the ability to reboot will depend on the state of the machines and interrupts. The hardware boards physically pull the machine down off their own onboard timers and will reboot from almost anything, we don't have a hardware board, but the software one did the trick. No more physical intervention required by shoji, the machine would reboot itself as needed. While this was going on Micah frantically searched for a cause and solution to this problem.
After many troubleshooting steps, Micah decided that it could be a hardware problem. He has seen this sort of behavior when there is bad memory involved. He got some new memory and ran
http://www.teresaudio.com/memtest86/ on it for 24 hours. Whenever you build a server it is
HIGHLY recommended that you test the memory before you deploy it! This was a learning experience. inglis had her memory tested before we put it into place, this is good practice. So he found some memory that passed the test and sent it out to replace the existing memory. This worked! Sarai no longer crashed.
It has been running fine for about 4 months now, but something has started to happen again to sarai. We had a problem where the power cord was bumped out, this was easily done because it didn't seem to seat securely, I thought that this was resolved, but it may not have been.
Thinking of putting
http://lwn.net/2001/0412/a/oom-patch.php3 on to the machine. We are stuck now because the remote machine seems to be OOM and I can't ssh in (sits there on debug: Connection established. and never continues), if I could set aside some memory for ssh, login, su, ps and kill, then I could remotely fix this problem.
--
MatRiseup - 21 Jun 2004 - Added statistics links