Traven - Mir installation

Scripted install

In order to avoid multiple tedious manual installs, there's now a script to do most of the steps:

/usr/local/sbin/mir-setup

System-level configuration for this script is in /etc/mir-setup/config, with environment variables (for Tomcat and Java) defined in /etc/mir-setup/env. It also uses an additional config file for a specific site, which is specified on the command line. See /etc/mir-setup/config-* for examples.

There are a few utility scripts in /usr/local/share/mir-setup which are required too. This stuff should all go into CVS at some point.

System Prerequisites

There are various bits of system configuration needed before the install works successfully.

Postgresql

By default, ident authentication is used -- this needs to be changed to password authentication for Mir to connect to the database. Nothing else on Traven is expected to use Postgresql, so I changed it for everything except the "postgres" user.

Tomcat

• TOMCAT_SECURITY turned off in /etc/default/tomcat4. I don't fully understand the implications of this, but recall Zapata saying that it's necessary. And with it turned on, Mir can't open many of its files.

• Debian gives Tomcat its own user, but not its own group. A dedicated group is useful for file permissions, so:

addgroup --system tomcat4
adduser tomcat4 tomcat4

• Also added a "tomcatstart" group, and this line in /etc/sudoers, to allow people to restart Tomcat without needing root:

%tomcatstart ALL=(root) NOPASSWD: /etc/init.d/tomcat4 restart

Apache

• Install mod_jk:

apt-get install libapache2-mod-jk2

and, since Debian have for some reason broken its default configuration:

cp /usr/share/doc/libapache2-mod-jk2/examples/workers2.properties.minimal /etc/apache2/workers2.properties

and patch /etc/apache2/workers2.properties thus:

@@ -7,7 +7,7 @@
 level=ERROR

 [config:]
-file=${serverRoot}/conf/workers2.properties
+file=${serverRoot}/workers2.properties
 debug=0
 debugEnv=0

then uncomment the JkSet line in /etc/apache2/mods-available/jk2.conf.

(Though I really don't understand why all this isn't done by default when the package is installed...)

• Disable logging of IP addresses, as per ApacheLogsWithoutIPs (using config directives for access log to change the "combined" format, and piping error log through a perl script -- /usr/local/bin/strip-ips-from-apache-error-log)

• Enable modules:

a2enmod ssl
a2enmod headers
a2enmod include
a2enmod deflate

• Disable these unless you need them for something else:

a2dismod perl
a2dismod php4
a2dismod python

• Enable port 443 for HTTPS (add "Listen 443" line to /etc/apache2/ports.conf

• Include Mir site config files:

# /etc/apache2/sites-available/mir-sites
Include /etc/apache2/mir-sites/*.conf

a2ensite mir-sites

• Include Mir site fragments on canonical SSL host (for SSL admin access without dedicated IPs, though note that the stylesheet will be missing for these):

# /etc/apache2/sites-available/mir-single-ssl-site
# Editor: vim:syn=apache
# SSL virtual host for all Mir sites
<VirtualHost traven.indymedia.org:443>
  ServerAdmin traven-tech@lists.indymedia.org

  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/traven.indymedia.org.crt
  SSLCertificateKeyFile /etc/apache2/ssl/traven.indymedia.org.key

  Include /etc/apache2/mir-sites/ssl-fragments/*.conf

  CustomLog /var/log/apache2/mir-single-ssl-site.log combined

  # Apache gzip compression
  <Location />
    # This is taken from here:
    # http://httpd.apache.org/docs-2.0/mod/mod_deflate.html
    # Insert filter
    SetOutputFilter DEFLATE
    # Netscape 4.x has some problems...
    BrowserMatch ^Mozilla/4 gzip-only-text/html
    # Netscape 4.06-4.08 have some more problems
    BrowserMatch ^Mozilla/4\.0[678] no-gzip
    # MSIE masquerades as Netscape, but it is fine
    BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
    # Don't compress images
    SetEnvIfNoCase Request_URI \
    \.(?:gif|jpe?g|png)$ no-gzip dont-vary
    # Make sure proxies don't deliver the wrong content
    Header append Vary User-Agent env=!dont-vary
  </Location>
</VirtualHost>

a2ensite mir-single-ssl-site

• Make sure the default virtual host (/etc/apache/sites-available/default) has the :80 port specifier, to match the virtual host definitions that will be generated later, ie:

NameVirtualHost *:80
<VirtualHost *:80>

rather than the current Debian default:

NameVirtualHost *
<VirtualHost *>

Java

• Blackdown JDK 1.4.2 installed: WARNING This is non-free software!

apt-get install j2sdk1.4

• Sun's JAI and JAI Image I/O libraries (CLASSPATH installs) downloaded from java.sun.com and untarred in /usr/local/lib (each creates its own subdirectory). The CLASSPATH and LD_LIBRARY_PATH settings required are in /etc/mir-setup/env. For the libraries to be found at runtime, the .jar and .so files are symlinked into /usr/share/tomcat4/common/lib.

• Sun's unlimited-strength crypto policy (JCE) installed. This will get overwritten if the JDK package is upgraded. I don't think we actually use the crypto stuff anyway though, just the SSL in Apache.

• (NOTE: You can now skip this section entirely, as Mir appears to run fine on Kaffe, which Debian installs by default when you install Tomcat. However, this appear to run significantly slower than Blackdown, by about a factor of ten.)

Other utilities that should be installed

• pwgen

apt-get install pwgen

• cvs

apt-get install cvs

Preparing to install a site

Before a site is installed, some user and group configuration needs to be done:

addgroup <sitename>
adduser --disabled-password --ingroup <sitename> <sitename>
addgroup <sitename>-tomcat
adduser <sitename> <sitename>-tomcat
adduser tomcat4 <sitename>-tomcat

Additionally, add all the users who are sysadmins for the site in question to the first group.

It is also useful to add the following entries to /etc/sudoers:

%<sitename>     ALL=(<sitename>) NOPASSWD: ALL
%<sitename>     ALL=(mir-setup) NOPASSWD: /usr/local/sbin/mir-setup --start <sitename>
%<sitename>     ALL=(mir-setup) NOPASSWD: /usr/local/sbin/mir-setup --stop <sitename>
%<sitename>     ALL=(mir-setup) NOPASSWD: /usr/local/sbin/mir-setup --reload <sitename>

Migrating an existing site onto Traven

• Copy the existing Mir config.properties file to /etc/mir-setup/config.properties-<sitename> on Traven. Replace explicit strings with setup variables like this:

Variable	Meaning	Exaxmple
$SITE	Short name of site	uk
$FQDN	Canonical fully-qualified domain name of non-SSL publishing/admin site	publish.indymedia.org.uk
$MIRRORFQDN	Canonical fully-qualified domain name of mirror site(s)	www.indymedia.org.uk
$SECUREFQDN	Canonical fully-qualified domain name of SSL publishing/admin site	publish.indymedia.org.uk
$PRODUCTIONDIR	Directory where produced files are stored in filesystem	/var/www/www.indymedia.org.uk
$DBNAME	Name of postgresql database	uk
$DBUSER	Postgresql user name	uk
$DBPASS	Postgresql password

These will all get replaced with the appropriate new values by the install script, even if you install multiple copies for testing, possibly on different servers. It will also take care of generating a random postgresql password.

• Create a mir-setup config file, /etc/mir-setup/config-<sitename>, using eg /etc/mir-setup/config-uk as an example.

• Take the existing publish server off-line, to prevent updates after the database is dumped (stop the webapp in Tomcat, remove the JkMount directives in httpd.conf or similar).

• Dump the existing database:

pg_dump -F t -d -x -u | gzip -9 >dump_file.tar.gz

(for postgres 7.1, as on kosmos; give database username and password at prompt); or

pg_dump -F t -d -x -U database_user | gzip -9 >dump_file.tar.gz

(for postgres 7.4, as on traven, give database password for specified user at prompt). The -d option (use INSERT rather than COPY) is vital unless you're migrating from exactly the same version of postgres (7.4.7 at the time of writing this, April 2005).

• Copy across the database dump to the location you specified in the config file.

• Run the install script:

/usr/local/sbin/mir-setup -f /etc/mir-setup/config-<sitename>

• Copy across robots.txt, include and media files from existing site:

scp -r user@oldserver:<DocumentRoot>/{images,icon,media,inc,robots.txt} /imc/<sitename>

or from a local copy on traven if you already have a mirror there. The above directories are correct for the UK site -- your setup may be different.

• Cease copying your site to live mirrors, so that they remain in their current intact state while the site regenerates.

• Change the DNS entry for your publish/admin site to point to traven.indymedia.org, preferably with a CNAME record in case traven's IP adress ever changes.

• Run Mir jobs (on your new site on traven) to generate site:
  • navigation.generate
  • stylesheets.generate
  • staticimages.generate
  • media.all
  • articles.all
  • startpage.generate
  • regionalstartpage.generate
  • topicstartpage.generate
  • regionaltopicstartpage.generate
  • syndication.generate
  • featurearchive.all
  • newswirearchive.all
  • viewallposts.all
  • regionalfeaturearchive.all
  • regionalnewswirearchive.all
  • regionalsyndication.generate
  • topicnewswirearchive.all
  • topicfeaturearchive.all
  • regionaltopicnewswirearchive.all
  • regionaltopicfeaturearchive.all

• You should now be up and running!

• Once you're satisfied that your site is working, resume copying to your live mirrors.

-- GarconDuMonde - 22 Mar 2005

-- ZaK - 10 Apr 2005