Actual set up - notes

Bios Settings

3ware bios

• staggered array
  • 2 second delay between spinup

Phoenix AwardBios

Limited CPUID MaxVal

Disabled

No-Execute Memory Protect

Enabled

hyper-Threading Technology

Enabled

Spread Spectrum

Disabled

DMI Event Log

Disabled

Console Redirection

Enabled (attempt to redirect console via COM port)

Baud rate

19200

Agent Connect via

Null

Agent wait time (min)

1

Boot devices

CDROM, Hard Disk, Legacy LAN

ACPI enabled

YES -> needed apparently for IPMI

Quick Boot and Quiet Boot

Enabled

Power on after power off

Disabled - best interest of colo

• method: ATA-6

POWER ON function

BUTTON ONLY

CPU warning temperature

75C/167F

Debian Settings

Locale

• USA

keymap

American English

Timezone

PST

nb: this was later changed (see below)

Partitioning

/ 15GB

• /dev/sda1
  • / 2GB

LVM 385GB

• volume group = vg
• physical volume = /dev/sda2
  • home 5.4 - ext3, nosuid, nodev
  • imc 214.7 - resierFS, noexec, nodev
  • mir 75.2 - ext3, nosuid, nodev
  • tmp 5.4 - ext3, encrypted with random keys, nosuid, nodev
  • upload 10.7 - ext3, noexec
  • var 21.5 - ext3, nosuid, nodev
  • postgres_data 20 - ext3, encrypted with gpg keyfile, noexec, nodev, noauto
  • swap 2.1 - encrypted with random keys

---

Interlude

partitioning done, thus go on to reboot... it still didn't work with the regular mode, but did work in single-user mode. wierd, as epsas says....

we progressed through:

• timezone set to GMT/UTC
• root password set
• first user set
• apt through http:
  • tried ualberta - failed
  • set to use debian.oregonstate.edu
  • core packages installed as per default (no "system" selected, i.e. web or mail or desktop or anything...)

---

IPMI

6 March 2005

• grabbed 2.6.10 deb kernel source.
• got diffs for 2.6.10 kernel for impi at http://sourceforge.net/project/showfiles.php?group_id=36127&package_id=110139 which is part of http://openipmi.sourceforge.net/
• followed steps for 2.6.8 kernel (substituting 10 for 8 as needed) at: ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/README-IPMI.htm
• copied 2.6.8 config to /usr/src/kernel-source-2.6.10
• make menuconfig:

Device Drivers ->
Character Devices-> 
IPMI ->

save config

vi Makefile    extraversion = -ipmi

make-kpkg --initrd -revision 01 kernel_image

• encountered build errors. first time, in DRM , so i deselected 3dfxvoodoo and one other item in that section, and restarted build.
• encountered fatal errors in the ipmi portion of build - undeclared variables and functions with wrong number of parameters.

- so

• got vanilla 2 6.10 kernel, patched with ipmi diffs built and installed
• observed that ipmi modules were in the lib/modules tree for current running kernel so tried those:

modprobe ipmi_devintf
modprobe ipmi_si
modprobe ipmi_msghandler
modprobe ipmi_watchdog

mknod /dev/ipmi0 c 254 0

• then tried ipmitool (version 1.6.0) (installed from deb obtained from surceforge site: http://sourceforge.net/project/showfiles.php?group_id=95200

• obtained only failures.
• IPMITool on Debian HowTo: http://buttersideup.com/docs/howto/IPMI_on_Debian.html

from the photo taken, we can see that our installed - presumable-ipmi-card is in the ipmi slot as described by the motherboard manual; the IPMI slot is horizontal white portion on right side of image; the 3ware card has the two SATA cables for the two drives, and takes up the left portion of the image below.

14 March 05

• set acpi in bios
• turned off oem boot logo
• the old "lose the screen on boot" returns.

• rebooting with stefani's 2.6.20 ipmi kernel
• not happy here: ipmitool still not working
  • loaded ipmi modules. ipmi_si fails. does not find interface.
• did mknod
• trying to modprobe ipmi_si again.
  • <stefani> the instructions do not quite match what i see

• <zak_work> i notice that the IPMI card is now showing up under ipmitool ... now have to learn what all the commands actually do...
  • http://buttersideup.com/docs/howto/IPMI_on_Debian.html
• <gdm> after stefani left last night, i ran this command:

 $ sudo modprobe ipmi_si type=kcs ports=0xca8 regspacings=4

• ... which seems to be what made it work

• from: ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/README-IPMI.htm
  • it's in the box "load the IPMI drivers" under option 2, point 3 b
  • other place it might have been found is at very bottom of that page above: question 7
    • Q. [IPMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. PMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. Why? Can I load the IPMI drivers manually?
• sounds like the settings should be provided by the BIOS, but they aren't being detected automatically. might be something to do with the "patch of handling IPMI registers with offsets" that's mentioned on that page... but i think we can live with manually specifying the module parameters. (They are now included in /etc/modprobe.d/ipmi-local, so they will be picked up automatically when you do "modprobe ipmi_si", and both ipmi_si and ipmi_devintf are automatically loaded from /etc/modules.)

16 March 2005

found this URL which is somewhat helpful:

http://buttersideup.com/docs/howto/IPMI_on_Debian.html

with that, i found in /usr/share/ipmitool the script bmclanconf

after bringing up eth1 on 69.901.34.249, i ran the script as such:

bmclanconf -c 1 -d -i eth1

and got the following output:

Auto-configuring eth1 (channel 1)
Setting LAN parameter macaddr 00:30:48:81:F7:11
ipmitool -I open lan set 1 macaddr 00:30:48:81:F7:11
Setting LAN parameter defgw ipaddr 69.90.134.129
ipmitool -I open lan set 1 defgw ipaddr 69.90.134.129
Setting LAN parameter defgw macaddr 00:0C:CE:B3:EF:00
ipmitool -I open lan set 1 defgw macaddr 00:0C:CE:B3:EF:00
Setting LAN parameter arp generate on
ipmitool -I open lan set 1 arp generate on
Setting LAN parameter arp interval 8
ipmitool -I open lan set 1 arp interval 8
Setting channel authentication capabilities
Setting LAN parameter auth callback,user,operator,admin md2,md5
ipmitool -I open lan set 1 auth callback,user,operator,admin md2,md5
Enabling channel 1
Setting LAN parameter access on
ipmitool -I open lan set 1 access on
Setting LAN parameter user
ipmitool -I open lan set 1 user

after getting things so that they seem to be right, one should be able to do something like this:

# IPMI_PASSWORD=< the ipmi password > ipmitool -I open -H 69.90.134.249 -E chassis status

Useful documents?

• IPMIView20.pdf: IPMI View User Guide

• Installation_20.pdf: SuperMicro IPMI 2.0 Solution Installation Guide

• IPMIcard.txt: README for IPMI 1.5/2.0 on Linux

---

Disk Encryption

• loopaes or dm-crypt?
  • loop-AES.README | Riseup page
  • dm-crypt | Riseup page
    • About swap encryption
• Comparison page on wiki.boum.org

• Currently running with loop-aes encrypted swap and /tmp (AES256, multi-key-v3, random keys).

• /var/lib/postgres/data now encrypted (AES256, multi-key-v3, gpg-encrypted keyfile). See TravenEncryptedPartitions for details of how these are managed.

one other thought - virtual servers?

• http://linux-vserver.org/Linux-VServer

Network Time

• http://www.tldp.org/HOWTO/TimePrecision-HOWTO/ntp.html

installed ntp and ntpdate using:

$ apt-get install ntp-simple ntpdate

then set /etc/ntp.conf to use the servers from pool.ntp.org.

$ ntpdate 0.pool.ntp.org
 9 Mar 13:45:07 ntpdate[25668]: adjust time server 192.36.143.151 offset -0.019599 sec
$

sshd_config

• http://www.debian.org/doc/manuals/reference/ch-tune.en.html#s-ssh-rsa
  • somewhere a long way down the page....!

PermitRootLogin no

PasswordAuthentication no

Mir and other software installation - take 1

Postgres

Setting up postgresql (7.4.7-2) ...

Creating config file /etc/postgresql/postmaster.conf with new version
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.

fixing permissions on existing directory /var/lib/postgres/data... ok
creating directory /var/lib/postgres/data/base... ok
creating directory /var/lib/postgres/data/global... ok
creating directory /var/lib/postgres/data/pg_xlog... ok
creating directory /var/lib/postgres/data/pg_clog... ok
selecting default max_connections... 100
selecting default shared_buffers... 1000
creating configuration files... ok
creating template1 database in /var/lib/postgres/data/base/1... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
creating information schema... ok
vacuuming database template1... ok
copying template1 to template0... ok

Success. The database server should be started automatically.
If not, you can start the database server using:

    /etc/init.d/postgresql start

Creating config file /etc/postgresql/postgresql.conf with new version

Apache-2

Setting up apache2-utils (2.0.53-5) ...
Setting up apache-utils (1.3.33-4) ...
Setting up ssl-cert (1.0-11) ...

Setting up apache2-common (2.0.53-5) ...
Setting Apache2 to Listen on port 80. If this is not desired, please edit /etc/apache2/ports.conf as desired. Note that the Port directive no longer works.
Module userdir installed; run /etc/init.d/apache2 force-reload to enable.

Setting up apache2-mpm-worker (2.0.53-5) ...
Starting web server: Apache2.

Setting up apache2 (2.0.53-5) ...

• nb: ChrisC posted the kosmos apache conf file in irc....

Tomcat4

Setting up tomcat4 (4.1.31-2) ...
Adding system user `tomcat4'...
Adding new user `tomcat4' (103) with group `nogroup'.
Not creating home directory.
Installing /var/lib/tomcat4/webapps/ROOT/WEB-INF/web.xml.
Installing /var/lib/tomcat4/conf/tomcat-users.xml.
Installing /var/lib/tomcat4/conf/jk2.properties
Starting Tomcat 4.1 servlet engine using Java from /usr/lib/kaffe: tomcat4.

Mir

Complex enough to have its own Wiki page: TravenMirInstall.

Mirrors

The set up of mirror sites is documented on TravenMirrors.

MTAs

Exim 4 is being used, from the Debian package. The Debian config is in monolithic-config-file mode -- ie /etc/exim4/exim4.conf.template is active, rather than /etc/exim4/conf.d. (Debian's config hacks to Exim 4 are quite extensive, and involve pre-processing the config file based on various parameters from debconf, which are stored in /etc/exim4/update-exim4.conf.conf)

• http://wiki.cacert.org/wiki/Exim4Configuration
• http://www.debian.org/doc/manuals/reference/ch-tune.en.html#s-mta
• http://www.exim.org/

Boot issues

• Getting this machine to boot successfully seems quite tricky. In particular, a hard power cycle (eg pulling out the mains cable) seems to be required. This is likely to mean that IPMI power-cycles are not sufficient, even once we have the remote management working.

• There appears to be a problem with APIC on this system. Adding "noapic" to the kernel command line seems to be necessary.

• Currently stefani's 2.6.10-ipmi kernel seems to be the most stable, although it doesn't have P4 optimisation or support for >1Gb RAM.

Network configuration

• This section has moved to TravenNetConfig

Remote management without IPMI

• Traven has a serial console on ttyS0/COM1 (9600 8N1), accessible from tsipoor ("conserver traven"). It's been tested once Linux is running; it should also work from GRUB but this hasn't been tested.

• No remote power control at present though.

Backups

Ideally traven should be set up like a debian push server and database backups should synced onto encrypted partitions.

Kosmos runs pg_backup.sh via cron:

00 03 * * * /usr/local/sbin/pg_backup.sh bva > /dev/null 2>&1

• pg_backup.sh.txt: Postgres backup script

See also BackupHowTo and Backupninja.

Traven Links

• TravenInfo
• TravenMirInstall
• TravenNetConfig

---

-- StefaniB - 07 Mar 2005
-- GarconDuMonde - 07 Mar 2005