Actual set up - notes
- staggered array
- 2 second delay between spinup
- Limited CPUID MaxVal
- No-Execute Memory Protect
- hyper-Threading Technology
- Spread Spectrum
- DMI Event Log
- Console Redirection
- Enabled (attempt to redirect console via COM port)
- Baud rate
- Agent Connect via
- Agent wait time (min)
- Boot devices
- CDROM, Hard Disk, Legacy LAN
- ACPI enabled
- YES -> needed apparently for IPMI
- Quick Boot and Quiet Boot
- Power on after power off
- Disabled - best interest of colo
- POWER ON function
- BUTTON ONLY
- CPU warning temperature
- American English
this was later changed (see below)
- volume group = vg
- physical volume = /dev/sda2
- home 5.4 - ext3, nosuid, nodev
- imc 214.7 - resierFS, noexec, nodev
- mir 75.2 - ext3, nosuid, nodev
- tmp 5.4 - ext3, encrypted with random keys, nosuid, nodev
- upload 10.7 - ext3, noexec
- var 21.5 - ext3, nosuid, nodev
- postgres_data 20 - ext3, encrypted with gpg keyfile, noexec, nodev, noauto
- swap 2.1 - encrypted with random keys
partitioning done, thus go on to reboot... it still didn't work with the regular mode, but did
work in single-user mode. wierd, as epsas says....
we progressed through:
- timezone set to GMT/UTC
- root password set
- first user set
- apt through http:
- tried ualberta - failed
- set to use debian.oregonstate.edu
- core packages installed as per default (no "system" selected, i.e. web or mail or desktop or anything...)
6 March 2005
Device Drivers ->
vi Makefile extraversion = -ipmi
make-kpkg --initrd -revision 01 kernel_image
- encountered build errors. first time, in DRM , so i deselected 3dfxvoodoo and one other item in that section, and restarted build.
- encountered fatal errors in the ipmi portion of build - undeclared variables and functions with wrong number of parameters.
- got vanilla 2 6.10 kernel, patched with ipmi diffs built and installed
- observed that ipmi modules were in the lib/modules tree for current running kernel so tried those:
mknod /dev/ipmi0 c 254 0
from the photo taken, we can see that our installed - presumable-ipmi-card is in the ipmi slot as described by the motherboard manual; the IPMI slot is horizontal white portion on right side of image; the 3ware card has the two SATA cables for the two drives, and takes up the left portion of the image below.
14 March 05
- set acpi in bios
- turned off oem boot logo
- the old "lose the screen on boot" returns.
- rebooting with stefani's 2.6.20 ipmi kernel
- not happy here: ipmitool still not working
- loaded ipmi modules. ipmi_si fails. does not find interface.
- did mknod
- trying to modprobe ipmi_si again.
- <stefani> the instructions do not quite match what i see
- <zak_work> i notice that the IPMI card is now showing up under ipmitool ... now have to learn what all the commands actually do...
- <gdm> after stefani left last night, i ran this command:
$ sudo modprobe ipmi_si type=kcs ports=0xca8 regspacings=4
- ... which seems to be what made it work
- from: ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/README-IPMI.htm
- it's in the box "load the IPMI drivers" under option 2, point 3 b
- other place it might have been found is at very bottom of that page above: question 7
- Q. [IPMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. PMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. Why? Can I load the IPMI drivers manually?
- sounds like the settings should be provided by the BIOS, but they aren't being detected automatically. might be something to do with the "patch of handling IPMI registers with offsets" that's mentioned on that page... but i think we can live with manually specifying the module parameters. (They are now included in /etc/modprobe.d/ipmi-local, so they will be picked up automatically when you do "modprobe ipmi_si", and both ipmi_si and ipmi_devintf are automatically loaded from /etc/modules.)
16 March 2005
found this URL which is somewhat helpful:
with that, i found in /usr/share/ipmitool the script bmclanconf
after bringing up eth1 on 69.901.34.249, i ran the script as such:
bmclanconf -c 1 -d -i eth1
and got the following output:
Auto-configuring eth1 (channel 1)
Setting LAN parameter macaddr 00:30:48:81:F7:11
ipmitool -I open lan set 1 macaddr 00:30:48:81:F7:11
Setting LAN parameter defgw ipaddr 18.104.22.168
ipmitool -I open lan set 1 defgw ipaddr 22.214.171.124
Setting LAN parameter defgw macaddr 00:0C:CE:B3:EF:00
ipmitool -I open lan set 1 defgw macaddr 00:0C:CE:B3:EF:00
Setting LAN parameter arp generate on
ipmitool -I open lan set 1 arp generate on
Setting LAN parameter arp interval 8
ipmitool -I open lan set 1 arp interval 8
Setting channel authentication capabilities
Setting LAN parameter auth callback,user,operator,admin md2,md5
ipmitool -I open lan set 1 auth callback,user,operator,admin md2,md5
Enabling channel 1
Setting LAN parameter access on
ipmitool -I open lan set 1 access on
Setting LAN parameter user
ipmitool -I open lan set 1 user
after getting things so that they seem to be right, one should be able to do something like this:
# IPMI_PASSWORD=< the ipmi password > ipmitool -I open -H 126.96.36.199 -E chassis status
- Currently running with loop-aes encrypted swap and /tmp (AES256, multi-key-v3, random keys).
- /var/lib/postgres/data now encrypted (AES256, multi-key-v3, gpg-encrypted keyfile). See TravenEncryptedPartitions for details of how these are managed.
one other thought - virtual servers
installed ntp and ntpdate using:
$ apt-get install ntp-simple ntpdate
then set /etc/ntp.conf
to use the servers from pool.ntp.org
$ ntpdate 0.pool.ntp.org
9 Mar 13:45:07 ntpdate: adjust time server 188.8.131.52 offset -0.019599 sec
Mir and other software installation - take 1
Setting up postgresql (7.4.7-2) ...
Creating config file /etc/postgresql/postmaster.conf with new version
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale C.
fixing permissions on existing directory /var/lib/postgres/data... ok
creating directory /var/lib/postgres/data/base... ok
creating directory /var/lib/postgres/data/global... ok
creating directory /var/lib/postgres/data/pg_xlog... ok
creating directory /var/lib/postgres/data/pg_clog... ok
selecting default max_connections... 100
selecting default shared_buffers... 1000
creating configuration files... ok
creating template1 database in /var/lib/postgres/data/base/1... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
creating information schema... ok
vacuuming database template1... ok
copying template1 to template0... ok
Success. The database server should be started automatically.
If not, you can start the database server using:
Creating config file /etc/postgresql/postgresql.conf with new version
Setting up apache2-utils (2.0.53-5) ...
Setting up apache-utils (1.3.33-4) ...
Setting up ssl-cert (1.0-11) ...
Setting up apache2-common (2.0.53-5) ...
Setting Apache2 to Listen on port 80. If this is not desired, please edit /etc/apache2/ports.conf as desired. Note that the Port directive no longer works.
Module userdir installed; run /etc/init.d/apache2 force-reload to enable.
Setting up apache2-mpm-worker (2.0.53-5) ...
Starting web server: Apache2.
Setting up apache2 (2.0.53-5) ...
- nb: ChrisC posted the kosmos apache conf file in irc....
Setting up tomcat4 (4.1.31-2) ...
Adding system user `tomcat4'...
Adding new user `tomcat4' (103) with group `nogroup'.
Not creating home directory.
Starting Tomcat 4.1 servlet engine using Java from /usr/lib/kaffe: tomcat4.
Complex enough to have its own Wiki page: TravenMirInstall
The set up of mirror sites is documented on TravenMirrors
Exim 4 is being used, from the Debian package. The Debian config is in monolithic-config-file mode -- ie /etc/exim4/exim4.conf.template is active, rather than /etc/exim4/conf.d. (Debian's config hacks to Exim 4 are quite extensive, and involve pre-processing the config file based on various parameters from debconf, which are stored in /etc/exim4/update-exim4.conf.conf)
- Getting this machine to boot successfully seems quite tricky. In particular, a hard power cycle (eg pulling out the mains cable) seems to be required. This is likely to mean that IPMI power-cycles are not sufficient, even once we have the remote management working.
- There appears to be a problem with APIC on this system. Adding "noapic" to the kernel command line seems to be necessary.
- Currently stefani's 2.6.10-ipmi kernel seems to be the most stable, although it doesn't have P4 optimisation or support for >1Gb RAM.
Remote management without IPMI
- Traven has a serial console on ttyS0/COM1 (9600 8N1), accessible from tsipoor ("conserver traven"). It's been tested once Linux is running; it should also work from GRUB but this hasn't been tested.
- No remote power control at present though.
Ideally traven should be set up like a debian push server
and database backups should synced onto encrypted partitions.
Kosmos runs pg_backup.sh via cron:
00 03 * * * /usr/local/sbin/pg_backup.sh bva > /dev/null 2>&1
See also BackupHowTo
- 07 Mar 2005
- 07 Mar 2005