Actual set up - notes
Bios Settings
3ware bios
- staggered array
- 2 second delay between spinup
Phoenix AwardBios
- Limited CPUID MaxVal
- Disabled
- No-Execute Memory Protect
- Enabled
- hyper-Threading Technology
- Enabled
- Spread Spectrum
- Disabled
- DMI Event Log
- Disabled
- Console Redirection
- Enabled (attempt to redirect console via COM port)
- Baud rate
- 19200
- Agent Connect via
- Null
- Agent wait time (min)
- 1
- Boot devices
- CDROM, Hard Disk, Legacy LAN
- ACPI enabled
- YES -> needed apparently for IPMI
- Quick Boot and Quiet Boot
- Enabled
- Power on after power off
- Disabled - best interest of colo
- POWER ON function
- BUTTON ONLY
- CPU warning temperature
- 75C/167F
Debian Settings
Locale
- keymap
- American English
- Timezone
- PST
nb: this was later changed (see below)
Partitioning
/ 15GB
LVM 385GB
- volume group = vg
- physical volume = /dev/sda2
- home 5.4 - ext3, nosuid, nodev
- imc 214.7 - resierFS, noexec, nodev
- mir 75.2 - ext3, nosuid, nodev
- tmp 5.4 - ext3, encrypted with random keys, nosuid, nodev
- upload 10.7 - ext3, noexec
- var 21.5 - ext3, nosuid, nodev
- postgres_data 20 - ext3, encrypted with gpg keyfile, noexec, nodev, noauto
- swap 2.1 - encrypted with random keys
Interlude
partitioning done, thus go on to reboot... it still didn't work with the regular mode, but
did work in single-user mode. wierd, as epsas says....
we progressed through:
- timezone set to GMT/UTC
- root password set
- first user set
- apt through http:
- tried ualberta - failed
- set to use debian.oregonstate.edu
- core packages installed as per default (no "system" selected, i.e. web or mail or desktop or anything...)
IPMI
6 March 2005
Device Drivers ->
Character Devices->
IPMI ->
save config
vi Makefile extraversion = -ipmi
make-kpkg --initrd -revision 01 kernel_image
- encountered build errors. first time, in DRM , so i deselected 3dfxvoodoo and one other item in that section, and restarted build.
- encountered fatal errors in the ipmi
portion of build - undeclared variables and functions with wrong number of parameters.
- so
- got vanilla 2 6.10 kernel, patched with ipmi diffs built and installed
- observed that ipmi modules were in the lib/modules tree for current running kernel so tried those:
modprobe ipmi_devintf
modprobe ipmi_si
modprobe ipmi_msghandler
modprobe ipmi_watchdog
mknod /dev/ipmi0 c 254 0
from the photo taken, we can see that our installed - presumable-ipmi-card is in the ipmi slot as described by the motherboard manual; the IPMI slot is horizontal white portion on right side of image; the 3ware card has the two SATA cables for the two drives, and takes up the left portion of the image below.
14 March 05
- set acpi in bios
- turned off oem boot logo
- the old "lose the screen on boot" returns.
- rebooting with stefani's 2.6.20 ipmi kernel
- not happy here: ipmitool still not working
- loaded ipmi modules. ipmi_si fails. does not find interface.
- did mknod
- trying to modprobe ipmi_si again.
- <stefani> the instructions do not quite match what i see
- <zak_work> i notice that the IPMI card is now showing up under ipmitool
... now have to learn what all the commands actually do...
- <gdm> after stefani left last night, i ran this command:
$ sudo modprobe ipmi_si type=kcs ports=0xca8 regspacings=4
- ... which seems to be what made it work
- from: ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/README-IPMI.htm
- it's in the box "load the IPMI drivers" under option 2, point 3 b
- other place it might have been found is at very bottom of that page above: question 7
- Q. [IPMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. PMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. Why? Can I load the IPMI drivers manually?
- sounds like the settings should be provided by the BIOS, but they aren't being detected automatically. might be something to do with the "patch of handling IPMI registers with offsets" that's mentioned on that page... but i think we can live with manually specifying the module parameters. (They are now included in /etc/modprobe.d/ipmi-local, so they will be picked up automatically when you do "modprobe ipmi_si", and both ipmi_si and ipmi_devintf are automatically loaded from /etc/modules.)
16 March 2005
found this URL which is somewhat helpful:
http://buttersideup.com/docs/howto/IPMI_on_Debian.html
with that, i found in /usr/share/ipmitool the script bmclanconf
after bringing up eth1 on 69.901.34.249, i ran the script as such:
bmclanconf -c 1 -d -i eth1
and got the following output:
Auto-configuring eth1 (channel 1)
Setting LAN parameter macaddr 00:30:48:81:F7:11
ipmitool -I open lan set 1 macaddr 00:30:48:81:F7:11
Setting LAN parameter defgw ipaddr 69.90.134.129
ipmitool -I open lan set 1 defgw ipaddr 69.90.134.129
Setting LAN parameter defgw macaddr 00:0C:CE:B3:EF:00
ipmitool -I open lan set 1 defgw macaddr 00:0C:CE:B3:EF:00
Setting LAN parameter arp generate on
ipmitool -I open lan set 1 arp generate on
Setting LAN parameter arp interval 8
ipmitool -I open lan set 1 arp interval 8
Setting channel authentication capabilities
Setting LAN parameter auth callback,user,operator,admin md2,md5
ipmitool -I open lan set 1 auth callback,user,operator,admin md2,md5
Enabling channel 1
Setting LAN parameter access on
ipmitool -I open lan set 1 access on
Setting LAN parameter user
ipmitool -I open lan set 1 user
after getting things so that they seem to be right, one should be able to do something like this:
# IPMI_PASSWORD=< the ipmi password > ipmitool -I open -H 69.90.134.249 -E chassis status
Useful documents?
Disk Encryption
- Currently running with loop-aes encrypted swap and /tmp (AES256, multi-key-v3, random keys).
- /var/lib/postgres/data now encrypted (AES256, multi-key-v3, gpg-encrypted keyfile). See TravenEncryptedPartitions for details of how these are managed.
one other thought -
virtual servers?
Network Time
installed ntp and ntpdate using:
$ apt-get install ntp-simple ntpdate
then set
/etc/ntp.conf to use the servers from
pool.ntp.org.
$ ntpdate 0.pool.ntp.org
9 Mar 13:45:07 ntpdate[25668]: adjust time server 192.36.143.151 offset -0.019599 sec
$
sshd_config
PermitRootLogin no
PasswordAuthentication no
Mir and other software installation - take 1
Postgres
Setting up postgresql (7.4.7-2) ...
Creating config file /etc/postgresql/postmaster.conf with new version
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale C.
fixing permissions on existing directory /var/lib/postgres/data... ok
creating directory /var/lib/postgres/data/base... ok
creating directory /var/lib/postgres/data/global... ok
creating directory /var/lib/postgres/data/pg_xlog... ok
creating directory /var/lib/postgres/data/pg_clog... ok
selecting default max_connections... 100
selecting default shared_buffers... 1000
creating configuration files... ok
creating template1 database in /var/lib/postgres/data/base/1... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
creating information schema... ok
vacuuming database template1... ok
copying template1 to template0... ok
Success. The database server should be started automatically.
If not, you can start the database server using:
/etc/init.d/postgresql start
Creating config file /etc/postgresql/postgresql.conf with new version
Apache-2
Setting up apache2-utils (2.0.53-5) ...
Setting up apache-utils (1.3.33-4) ...
Setting up ssl-cert (1.0-11) ...
Setting up apache2-common (2.0.53-5) ...
Setting Apache2 to Listen on port 80. If this is not desired, please edit /etc/apache2/ports.conf as desired. Note that the Port directive no longer works.
Module userdir installed; run /etc/init.d/apache2 force-reload to enable.
Setting up apache2-mpm-worker (2.0.53-5) ...
Starting web server: Apache2.
Setting up apache2 (2.0.53-5) ...
- nb: ChrisC posted the kosmos apache conf file in irc....
Tomcat4
Setting up tomcat4 (4.1.31-2) ...
Adding system user `tomcat4'...
Adding new user `tomcat4' (103) with group `nogroup'.
Not creating home directory.
Installing /var/lib/tomcat4/webapps/ROOT/WEB-INF/web.xml.
Installing /var/lib/tomcat4/conf/tomcat-users.xml.
Installing /var/lib/tomcat4/conf/jk2.properties
Starting Tomcat 4.1 servlet engine using Java from /usr/lib/kaffe: tomcat4.
Mir
Complex enough to have its own Wiki page:
TravenMirInstall.
Mirrors
The set up of mirror sites is documented on
TravenMirrors.
MTAs
Exim 4 is being used, from the Debian package. The Debian config is in monolithic-config-file mode -- ie /etc/exim4/exim4.conf.template is active, rather than /etc/exim4/conf.d. (Debian's config hacks to Exim 4 are quite extensive, and involve pre-processing the config file based on various parameters from debconf, which are stored in /etc/exim4/update-exim4.conf.conf)
Boot issues
- Getting this machine to boot successfully seems quite tricky. In particular, a hard power cycle (eg pulling out the mains cable) seems to be required. This is likely to mean that IPMI power-cycles are not sufficient, even once we have the remote management working.
- There appears to be a problem with APIC on this system. Adding "noapic" to the kernel command line seems to be necessary.
- Currently stefani's 2.6.10-ipmi kernel seems to be the most stable, although it doesn't have P4 optimisation or support for >1Gb RAM.
Network configuration
Remote management without IPMI
- Traven has a serial console on ttyS0/COM1 (9600 8N1), accessible from tsipoor ("conserver traven"). It's been tested once Linux is running; it should also work from GRUB but this hasn't been tested.
- No remote power control at present though.
Backups
Ideally traven should be set up like a
debian push server and database backups should synced onto encrypted partitions.
Kosmos runs pg_backup.sh via cron:
00 03 * * * /usr/local/sbin/pg_backup.sh bva > /dev/null 2>&1
See also
BackupHowTo and
Backupninja.
Traven Links
--
StefaniB - 07 Mar 2005
--
GarconDuMonde - 07 Mar 2005