Actual set up - notes

Bios Settings

3ware bios

  • staggered array
    • 2 second delay between spinup

Phoenix AwardBios

Limited CPUID MaxVal
Disabled
No-Execute Memory Protect
Enabled
hyper-Threading Technology
Enabled
Spread Spectrum
Disabled

DMI Event Log
Disabled

Console Redirection
Enabled (attempt to redirect console via COM port)
Baud rate
19200
Agent Connect via
Null
Agent wait time (min)
1

Boot devices
CDROM, Hard Disk, Legacy LAN

ACPI enabled
YES -> needed apparently for IPMI
Quick Boot and Quiet Boot
Enabled
Power on after power off
Disabled - best interest of colo
  • method: ATA-6

POWER ON function
BUTTON ONLY

CPU warning temperature
75C/167F

Debian Settings

Locale

  • USA
keymap
American English
Timezone
PST

nb: this was later changed (see below)

Partitioning

/ 15GB

  • /dev/sda1
    • / 2GB

LVM 385GB

  • volume group = vg
  • physical volume = /dev/sda2
    • home 5.4 - ext3, nosuid, nodev
    • imc 214.7 - resierFS, noexec, nodev
    • mir 75.2 - ext3, nosuid, nodev
    • tmp 5.4 - ext3, encrypted with random keys, nosuid, nodev
    • upload 10.7 - ext3, noexec
    • var 21.5 - ext3, nosuid, nodev
    • postgres_data 20 - ext3, encrypted with gpg keyfile, noexec, nodev, noauto
    • swap 2.1 - encrypted with random keys


Interlude

partitioning done, thus go on to reboot... it still didn't work with the regular mode, but did work in single-user mode. wierd, as epsas says....

we progressed through:

  • timezone set to GMT/UTC
  • root password set
  • first user set
  • apt through http:
    • tried ualberta - failed
    • set to use debian.oregonstate.edu
    • core packages installed as per default (no "system" selected, i.e. web or mail or desktop or anything...)


IPMI

6 March 2005

Device Drivers ->
Character Devices-> 
IPMI ->

save config

vi Makefile    extraversion = -ipmi

make-kpkg --initrd -revision 01 kernel_image

  • encountered build errors. first time, in DRM , so i deselected 3dfxvoodoo and one other item in that section, and restarted build.
  • encountered fatal errors in the ipmi frown, sad smile portion of build - undeclared variables and functions with wrong number of parameters.

- so

  • got vanilla 2 6.10 kernel, patched with ipmi diffs built and installed
  • observed that ipmi modules were in the lib/modules tree for current running kernel so tried those:

modprobe ipmi_devintf
modprobe ipmi_si
modprobe ipmi_msghandler
modprobe ipmi_watchdog

mknod /dev/ipmi0 c 254 0

from the photo taken, we can see that our installed - presumable-ipmi-card is in the ipmi slot as described by the motherboard manual; the IPMI slot is horizontal white portion on right side of image; the 3ware card has the two SATA cables for the two drives, and takes up the left portion of the image below.

IPMI

14 March 05

  • set acpi in bios
  • turned off oem boot logo
  • the old "lose the screen on boot" returns.

  • rebooting with stefani's 2.6.20 ipmi kernel
  • not happy here: ipmitool still not working
    • loaded ipmi modules. ipmi_si fails. does not find interface.
  • did mknod
  • trying to modprobe ipmi_si again.
    • <stefani> the instructions do not quite match what i see

 $ sudo modprobe ipmi_si type=kcs ports=0xca8 regspacings=4

  • ... which seems to be what made it work

  • from: ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/README-IPMI.htm
    • it's in the box "load the IPMI drivers" under option 2, point 3 b
    • other place it might have been found is at very bottom of that page above: question 7
      • Q. [IPMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. PMI 2.0] I've upgraded the Linux kernel and IPMI drivers, but I still cannot load the IPMI drivers automatically. Why? Can I load the IPMI drivers manually?
  • sounds like the settings should be provided by the BIOS, but they aren't being detected automatically. might be something to do with the "patch of handling IPMI registers with offsets" that's mentioned on that page... but i think we can live with manually specifying the module parameters. (They are now included in /etc/modprobe.d/ipmi-local, so they will be picked up automatically when you do "modprobe ipmi_si", and both ipmi_si and ipmi_devintf are automatically loaded from /etc/modules.)

16 March 2005

found this URL which is somewhat helpful:

http://buttersideup.com/docs/howto/IPMI_on_Debian.html

with that, i found in /usr/share/ipmitool the script bmclanconf

after bringing up eth1 on 69.901.34.249, i ran the script as such:

bmclanconf -c 1 -d -i eth1

and got the following output:

Auto-configuring eth1 (channel 1)
Setting LAN parameter macaddr 00:30:48:81:F7:11
ipmitool -I open lan set 1 macaddr 00:30:48:81:F7:11
Setting LAN parameter defgw ipaddr 69.90.134.129
ipmitool -I open lan set 1 defgw ipaddr 69.90.134.129
Setting LAN parameter defgw macaddr 00:0C:CE:B3:EF:00
ipmitool -I open lan set 1 defgw macaddr 00:0C:CE:B3:EF:00
Setting LAN parameter arp generate on
ipmitool -I open lan set 1 arp generate on
Setting LAN parameter arp interval 8
ipmitool -I open lan set 1 arp interval 8
Setting channel authentication capabilities
Setting LAN parameter auth callback,user,operator,admin md2,md5
ipmitool -I open lan set 1 auth callback,user,operator,admin md2,md5
Enabling channel 1
Setting LAN parameter access on
ipmitool -I open lan set 1 access on
Setting LAN parameter user
ipmitool -I open lan set 1 user

after getting things so that they seem to be right, one should be able to do something like this:

# IPMI_PASSWORD=< the ipmi password > ipmitool -I open -H 69.90.134.249 -E chassis status

Useful documents?


Disk Encryption

  • Currently running with loop-aes encrypted swap and /tmp (AES256, multi-key-v3, random keys).

  • /var/lib/postgres/data now encrypted (AES256, multi-key-v3, gpg-encrypted keyfile). See TravenEncryptedPartitions for details of how these are managed.

one other thought - virtual servers?

Network Time

installed ntp and ntpdate using:

$ apt-get install ntp-simple ntpdate

then set /etc/ntp.conf to use the servers from pool.ntp.org.

$ ntpdate 0.pool.ntp.org
 9 Mar 13:45:07 ntpdate[25668]: adjust time server 192.36.143.151 offset -0.019599 sec
$

sshd_config

PermitRootLogin no

PasswordAuthentication no

Mir and other software installation - take 1

Postgres

Setting up postgresql (7.4.7-2) ...

Creating config file /etc/postgresql/postmaster.conf with new version
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.

fixing permissions on existing directory /var/lib/postgres/data... ok
creating directory /var/lib/postgres/data/base... ok
creating directory /var/lib/postgres/data/global... ok
creating directory /var/lib/postgres/data/pg_xlog... ok
creating directory /var/lib/postgres/data/pg_clog... ok
selecting default max_connections... 100
selecting default shared_buffers... 1000
creating configuration files... ok
creating template1 database in /var/lib/postgres/data/base/1... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
creating information schema... ok
vacuuming database template1... ok
copying template1 to template0... ok

Success. The database server should be started automatically.
If not, you can start the database server using:

    /etc/init.d/postgresql start

Creating config file /etc/postgresql/postgresql.conf with new version

Apache-2

Setting up apache2-utils (2.0.53-5) ...
Setting up apache-utils (1.3.33-4) ...
Setting up ssl-cert (1.0-11) ...

Setting up apache2-common (2.0.53-5) ...
Setting Apache2 to Listen on port 80. If this is not desired, please edit /etc/apache2/ports.conf as desired. Note that the Port directive no longer works.
Module userdir installed; run /etc/init.d/apache2 force-reload to enable.

Setting up apache2-mpm-worker (2.0.53-5) ...
Starting web server: Apache2.

Setting up apache2 (2.0.53-5) ...

  • nb: ChrisC posted the kosmos apache conf file in irc....

Tomcat4

Setting up tomcat4 (4.1.31-2) ...
Adding system user `tomcat4'...
Adding new user `tomcat4' (103) with group `nogroup'.
Not creating home directory.
Installing /var/lib/tomcat4/webapps/ROOT/WEB-INF/web.xml.
Installing /var/lib/tomcat4/conf/tomcat-users.xml.
Installing /var/lib/tomcat4/conf/jk2.properties
Starting Tomcat 4.1 servlet engine using Java from /usr/lib/kaffe: tomcat4.

Mir

Complex enough to have its own Wiki page: TravenMirInstall.

Mirrors

The set up of mirror sites is documented on TravenMirrors.

MTAs

Exim 4 is being used, from the Debian package. The Debian config is in monolithic-config-file mode -- ie /etc/exim4/exim4.conf.template is active, rather than /etc/exim4/conf.d. (Debian's config hacks to Exim 4 are quite extensive, and involve pre-processing the config file based on various parameters from debconf, which are stored in /etc/exim4/update-exim4.conf.conf)

Boot issues

  • Getting this machine to boot successfully seems quite tricky. In particular, a hard power cycle (eg pulling out the mains cable) seems to be required. This is likely to mean that IPMI power-cycles are not sufficient, even once we have the remote management working.

  • There appears to be a problem with APIC on this system. Adding "noapic" to the kernel command line seems to be necessary.

  • Currently stefani's 2.6.10-ipmi kernel seems to be the most stable, although it doesn't have P4 optimisation or support for >1Gb RAM.

Network configuration

Remote management without IPMI

  • Traven has a serial console on ttyS0/COM1 (9600 8N1), accessible from tsipoor ("conserver traven"). It's been tested once Linux is running; it should also work from GRUB but this hasn't been tested.

  • No remote power control at present though.

Backups

Ideally traven should be set up like a debian push server and database backups should synced onto encrypted partitions.

Kosmos runs pg_backup.sh via cron:

00 03 * * * /usr/local/sbin/pg_backup.sh bva > /dev/null 2>&1

See also BackupHowTo and Backupninja.


-- StefaniB - 07 Mar 2005
-- GarconDuMonde - 07 Mar 2005
Topic attachments
I Attachment Action Size Date Who Comment
IPMIView20.pdfpdf IPMIView20.pdf manage 955 K 07 Mar 2005 - 15:07 GarconDuMonde IPMI View User Guide
IPMIcard.txttxt IPMIcard.txt manage 12 K 07 Mar 2005 - 15:13 GarconDuMonde README for IPMI 1.5/2.0 on Linux
Installation_20.pdfpdf Installation_20.pdf manage 441 K 07 Mar 2005 - 15:08 GarconDuMonde SuperMicro IPMI 2.0 Solution Installation Guide
our_ipmi.jpgjpg our_ipmi.jpg manage 15 K 07 Mar 2005 - 05:53 UnknownUser  
pg_backup.sh.txttxt pg_backup.sh.txt manage 16 K 19 Apr 2005 - 11:38 UnknownUser Postgres backup script
Topic revision: r26 - 02 Jun 2005, GarconDuMonde
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback