Upto UkDevServer | ImcUkTech

UK Mir Development Sites

Quite a lot of the following is out of date — there is now a script for setting up dev sites, see PulpInfo

The plan is to have one install of Mir per developer. These are are instructions for setting one up.

The first (and most important) part of this document is based on the install of cambridge.dev.indymedia.org.uk. The complementary section is based on the more recent installation of oxford.dev.indymedia.org.uk. See also the UkDev2 install notes from setting up dev2.indymedia.org.uk.

Before you get too excited

In order to follow the procedure below you will need to have the access passwords (including root) for the development server, a valid account on codecoop (see below) and a valid Mir account for the development database (unless you fancy setting up another database, which you are welcome to). Furthermore you will have to be a member of the imc-uk project, and upload your gpg public key(s) to codecoop. Everything is explained below but expect some of this to take time. Therefore you should make sure you are fully set up well before your collective really needs to make modifications to your site.

Accounts and dirs

You need root to set all this up. First create a directory for the site and chown it to the users name:

# mkdir /var/www/cambridge.dev.indymedia.org.uk
# chown manos.manos /var/www/cambridge.dev.indymedia.org.uk

Also the directories where Mir will write the site needs creating and chowning, and a symlink creating for index.html:

# cd /var/www/cambridge.dev.indymedia.org.uk
# mkdir -p site/{,abstract,comments,content,de,en,img,inc,style}
# chown tomcat4. site/{,abstract,comments,content,de,en,img,inc,style}
# cd site
# ln -s en/index.html .

And you should put a fairly restrictive robots.txt in there, for example by copying from another installation:

# cp /var/www/dev.indymedia.org.uk/site/robots.txt /var/www/cambridge.dev.indymedia.org.uk/site/

Mir checkout

Then as the user that owns the directory do an anonymous checkout of Mir:

$ cd /var/www/cambridge.dev.indymedia.org.uk
$ cvs -d:pserver:anonymous@cvs.codecoop.org:/cvsroot/mir login
$ cvs -z3 -d:pserver:anonymous@cvs.codecoop.org:/cvsroot/mir co -r MIR_1_1 mir

UK checkout

There are two ways you can check out the imc-uk project: as an anonymous user, which means you won't be able to edit files on the dev server and then commit them; or as a registered codecoop user in which case you will be able to commit then back to codecoop so that they can propagate to the real IMC-UK site.

Anonymous

$ cd /var/www/cambridge.dev.indymedia.org.uk
$ cvs -d:pserver:anonymous@cvs.codecoop.org:/cvsroot/imc-uk login
$ cvs -z3 -d:pserver:anonymous@cvs.codecoop.org:/cvsroot/imc-uk co setup_uk

When prompted for a password press Enter.

Using a codecoop account

If you have a user account on codecoop and you have been added to the imc-uk project then you can check out the templates with write premissions.

To get added to the imc-uk project ask in #uk on irc or on imc-uk-tech.

Preparing your codecoop account

For this to work you first need to upload a ssh public key to your codecoop account, to do this first you need to generate a key pair. It is best to have a seperate key pair for use with the dev server / codecoop from the key pair you will also want to set up with your desktop machine.

There are two ways of working and you should decide first which you will use (of course you can do both).

Editing templates locally

If you want to be able to edit the templates on your desktop machine and then commit them to codecoop then you need to check the templates out over ssh in order to have write permissions to upload them again. For this you need to generate you ssh key pair on your desktop machine.

If you don't generate a key pair on the development machine then you will have to scp the edited template from your desktop machine to the development machine in order to test them.

If you generate a ssh key pair on the development machine then you can update the templates on this machine direct from codecoop, if you don't want to do this skip the next section.

Editing templates remotly

If you want to edit the templates on the development machine and then commit them to codecoop from there then you will need to generate a keypair on the development machine. This is because you don't want to put the private key from your desktop machine onto the development machine.

Generating a ssh key pair

$ cd ~
$ ssh-keygen -t rsa
Enter file in which to save the key (/home/manos/.ssh/id_rsa): (here you can hit enter)

When prompted for a passphrase you should use a new password and not the same one as any other you have.

Uploading your ssh public key to codecoop

Then you need to upload the public key, id_ras.pub, to codecoop. To submit your key login into codecoop.org, then browse to your personal page (https://codecoop.org/my/). At "Account Maintanance" you can edit your account (https://codecoop.org/account/). On the bottom of this page is a link to edit your ssh keys (https://codecoop.org/account/editsshkeys.php). Just paste your public key into the form and submit. After about one hour you are able to connect to codecoop.org via ssh.

Take EXTRA care with whitespace when copying ssh keys!

Mozex is a Mozilla / Firefox plugin that allows external editors to be used for editing text areas, using this can help with the whitespace problem.

If you don't have an account on codecoop you should create one and then ask in #uk on irc.indymedia.org or the imc-uk-tech list to be added to the project.

Check out the template over ssh

To check out the UK stuff from CVS using SSH (you will need to change the usename from manos to your username!):

$ cd /var/www/cambridge.dev.indymedia.org.uk
$ export CVS_RSH=ssh
$ cvs -z3 -d:ext:manos@cvs.codecoop.org:/cvsroot/imc-uk co setup_uk

ssh see also

The global tech SshHowTo

For a general article on setting up ssh keys on Fedora see Howto setup SSH keys between machines - these instructions are good for setting it up so you can login to the development machine via ssh without a password. But you will still need to type a passphrase each time unless you set up ssh-agent, Red Hat have some instructions for Configuring ssh-agent with GNOME.

Apache

(You need to be "root" to do the stuff in this section) You can copy an existing apache config and edit the paths as needed, for example:

# cd /etc/httpd/conf
# cp dev.conf ../conf.d/cambridge.conf

After editing the config file should look like this:

# Editor: vim:syn=apache
# cambridge.dev.indymedia.org.uk
<VirtualHost *:80>
 ServerAdmin imc-uk-tech@lists.indymedia.org
 DocumentRoot /var/www/cambridge.dev.indymedia.org.uk/site
 ServerName cambridge.dev.indymedia.org.uk
<Location /cambridge>
 JkUriSet worker ajp13:localhost:8009
</Location>
<Location /cambridge/*>
 JkUriSet worker ajp13:localhost:8009
</Location>
<Directory /var/www/cambridge.dev.indymedia.org.uk/site>
 Options Indexes IncludesNoExec FollowSymLinks MultiViews    
 IndexOptions +SuppressHTMLPreamble +SuppressDescription    
 AddHandler server-parsed .html    
 AddType 'text/html; charset=utf-8' .html
 AllowOverride None
 Order allow,deny 
 Allow from all  
 </Directory>
 # Apache gzip compression
<Location />
 # This is taken from here: 
 # http://httpd.apache.org/docs-2.0/mod/mod_deflate.html
 # Insert filter
 SetOutputFilter DEFLATE
 # Netscape 4.x has some problems...
 BrowserMatch ^Mozilla/4 gzip-only-text/html
 # Netscape 4.06-4.08 have some more problems
 BrowserMatch ^Mozilla/4\.0[678] no-gzip
 # MSIE masquerades as Netscape, but it is fine
 BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 # Don't compress images
 SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png)$ no-gzip dont-vary
 # Make sure proxies don't deliver the wrong content
 Header append Vary User-Agent env=!dont-vary
</Location>
</VirtualHost>

The file gets included automatically after restart, so do a config test and then restart apache:

# /etc/init.d/httpd configtest
Syntax OK
# /etc/init.d/httpd graceful

Copy the uk setup into mir

You need to replace the etc folder in the mir installation with the corresponding etc directory in the setup_uk installation:

$ cp -r setup_uk/etc mir/.

Mir configuration

You will need to configure mir so that it knows which folder things are in. Copy the config.properties file into your mir instalation, and edit it.

$ cp /var/www/dev.indymedia.org.uk/mir/etc/config.properties mir/etc/config.properties 

Edit the file and change all occurances of "dev.indymedia.org.uk" to cambridge.dev.indymedia.org.uk". Also change "dev.indymedia.org.uk/dev" to "cambridge.dev.indymedia.org.uk/cambridge"

Compile mir

Just go into mir's directory and compile it using Ant:

$ cd mir
$ ant

This should take less than a minute.

Filesysytem permissions

Some of the files in the mir directory need to be given the correct permissions, and there is a handy script provided to do this. There is also a directory which is likely to be missing:

# cd /var/www/cambridge.dev.indymedia.org.uk/mir
# mkdir bin/mir/WEB-INF/log
# cp perms.sh-dist perms.sh

Edit this file, changing the GROUP setting near the top to:

GROUP=tomcat4

And then run the script:

./perms.sh

Tomcat link

Create a symbolic link from the tomcat web applications directory to your bin directory: (You need to be root to do that!)

# cd /var/lib/tomcat4/webapps/
# ln -s /var/www/cambridge.dev.indymedia.org.uk/mir/bin/mir cambridge

To restart tomcat:

# /etc/init.d/tomcat4 restart

(You will need to re-run ant and restart tomcat after any future changes to config.properties before they will take effect.)

Generating your site

If all is well, you should now have a working installation of Mir, but the actual HTML pages for your site haven't been generated yet. Go to http://cambridge.dev.indymedia.org.uk/cambridge and log in using your dev admin account.

Then go to the advanced page and run the jobs one by one. Begin with staticimages.generate and media.all, then navigation.generate, stylesheets.generate, articles.sample and startpage.generate. These should be enough to verify that things are working. Once they have completed, go to http://cambridge.dev.indymedia.org.uk/ and check that the front page of the site appears.

Now you can go back to the advanced page and generate everything else (articles.all, regions, topics, archives etc). Note that you don't need to run synchronisation jobs on the dev server, as there is no separation between the publishing server and the "live" server.

Includes

There are various SSI included files that can be be edited via the admin interface, to set these up you need to touch the files:

  $ cd /var/www/dev2.indymedia.org.uk/site/inc
  $ su
  # touch events.inc
  # chown tomcat4.tomcat4 events.inc 

There is a script for doing this that saves time: setup_uk/misc/touch_includes.sh use it like this:

  $ cd /var/www/dev2.indymedia.org.uk/site/inc
  $ su
  # cp ../../setup_uk/misc/touch_includes.sh .
  # sh ./touch_includes.sh 
  # chown -R tomcat4.tomcat4 *
  # rm touch_includes.sh

IMCUKAdminLocalizer.java

You might not need to do this, the [[][install of dev2.indymedia.org.uk]] didn't need to have it done, so skip it and come back if there there is a problem.

The latest Mir requires this change, remove the following line from ./etc/extrasource/IMCUKAdminLocalizer.java

  ModuleArticleType module = new ModuleArticleType(DatabaseArticleType.getInstance());

And add:

  ModuleArticleType module = new ModuleArticleType();

Troubleshooting

Please flesh this section out if you experience any difficulties!

The following log files are likely to be useful in tracking things down:

• /var/www/cambridge.dev.indymedia.org.uk/mir/bin/mir/WEB-INF/log/* (Mir's log files)
• /var/log/tomcat4/catalina.out (Tomcat log, including Java exceptions from within Mir)
• /var/log/httpd/error_log (Apache error log)
• /var/log/httpd/access_log (Apache access log)

---

Complementary Instructions on How to Setup a Regional Dev Site

I just finished setting up the Oxford development installation on the development server. I will write here what exactly I had to do, as a more basic complement to the above instructions. If you are slightly lost with the instructions above, just follow this step-by-step guide!

Preliminaries

• You need to get a SSH key. First, SSH must be installed on your computer. On Linux it should be there already. Then follow the instructions above on how to create an SSH key.
• You need an account on the CVS server, http://codecoop.org . You can create it yourself. Then you have to ask someone (say chrisc) on irc.indymedia.org, channel #uk, to make you part of the imc-uk project on codecoop so that you have write permissions to the templates.
• Once it's done, you should upload your SSH public key to the codecoop server. Follow the instructions above on how to do that.
• You need to install CVS on your computer (CVS is a way of keeping track of the changes made to the files, by giving them version numbers rather than deleting the old files. That way you can always go back to the previous file if you don't like the changes you've made). With my Ubuntu/Debian box, all I had to do was "apt-get install cvs".
• You need an account on the development server, pulp.indymedia.org.uk . You have to ask someone (say chrisc) on irc.indymedia.org, channel #uk, to make one for you. You'll have to provide your SSH public key so that you can login to the dev server with SSH.

OK, after all these preliminaries you are almost ready to start playing with the templates!

Creating your own MIR installation on the dev server

Now, you should create your own MIR installation on the dev server, so that you can play with it as much as you want without interfering with other people's modifications. It is explained above how to do that. However, you need root permissions to the dev server to do that, so you will need help from people with root access. Actually, zak wrote a script that does the installation automatically, rather than going through the steps explained above. So you should go on irc and ask zak to help you setting up your own MIR installation.

Then you will have a Mir installation at, say, http://oxford.dev.indymedia.org.uk/. The admin section will be at http://oxford.dev.indymedia.org.uk/oxford. You will need to ask zak or someone else to create an admin user account for you (and/or your collective) on your dev installation, so that you can login to the admin section.

Modifying the templates using CVS

Checkout the files

Now that you've done all that, you probably want to get the files and start working on them. For this you need to use CVS. The first thing you must do is "checkout" the files on your computer. If you uploaded your SSH key to codecoop, and that you are a registered developer of the imc-uk project on codecoop, you can checkout the files with write permission, which means that you will be able to save your changes on the codecoop server. To do that, if you are on Linux, go into the folder in which you want to checkout the templates and in a shell type (change societelibre for your codecoop username):

export CVS_RSH=ssh
export CVSROOT=":ext:societelibre@cvs.codecoop.org:/var/lib/cvs"
cvs -z3 -d:ext:societelibre@cvs.codecoop.org:/cvsroot/imc-uk co setup_uk

You now have the whole UK Indymedia tree on your computer!

Add or Remove Files

If you want to add (remove) files on the codecoop server, you must do the following:

For non-binary files:

cvs add (remove) file_name.html

For binary files (like pictures):

cvs add (remove) -kb file_name.png

Commit your changes

Now you can modify the templates on your computer. When you are finished, you must commit your changes to the CVS server. To do that, type

cvs ci

Then a new version of the files you changed will be created (you will have to comment it to say what changes you made).

Before modifying the files

Before you modify the files on your computer, always make sure you have the newest version of the files by doing an update

cvs update

Update the files on your dev server

OK, now you've modified the files, and it's been committed to the CVS codecoop server. Now you would like to see the changes online! So you have to update the files on your, say, Oxford dev server. Here is how you do that.

You must first access the dev server using ssh (change societelibre for your username):

ssh societelibre@pulp.indymedia.org.uk

Then go to the setup_uk directory and update your changes:

cd /var/www/oxford.dev.indymedia.org.uk/setup_uk
cvs update

Now you have to recompile by doing:

cd /var/www/oxford.dev.indymedia.org.uk/mir
ant

Finally, restart tomcat:

sudo /etc/init.d/tomcat4 restart

There it is! The files have been updated. The only remaining thing to do, as with the actual UK Indymedia site, is to regenerate the modified files from the admin interface. So go to the admin interface, login, go to the advanced page, and regenerate what has been modified.

Now your changes should be online at http://oxford.dev.indymedia.org.uk!

Modifying the CSS stylesheet from the admin interface

There is now a very nice hack, due to manos, that allows to modify the CSS stylesheet directly from the MIR admin interface, through the "edit include files" option. I will now try to explain how to implement this hack on your regional site.

Include files

First, you must modify the includes through CVS. As explained above in the includes section, there is a script to do that. You must first modify this script. After having cvs updated on your computer, go to /setup_uk/misc and edit the touch_includes.sh script. You must change you regional line by (for oxford):

touch {oxford/events.inc,oxford/links.inc,oxford/banners.inc,oxford/style.css}

This will create two new include files, banners.inc and style.css, that can be modified from the admin interface. Now you have to cvs ci your changes, and as explained above, ssh to the dev server and update your changes to your regional dev installation.

Now you must tell MIR to look for .css files when it looks for include files. I am not quite sure that this is the best way to do this (i mean how to do this change). But what you have to do is to change the file on the dev server /var/www/oxford.dev.indymedia.org.uk/setup_uk/etc/config.properties. You can do that by ssh to the server and modify the file (using say the text editor nano), but i'm not sure that is the good way to do it (we should somehow use CVS shouldn't we?). Anyway, what you must do is replace the line

ServletModule.FileEdit.Configuration= \
  includes:/var/www/oxford.dev.indymedia.org.uk/site/inc:.*\.inc:1

by

ServletModule.FileEdit.Configuration= \
  includes:/var/www/oxford.dev.indymedia.org.uk/site/inc:.*\.(inc|css):1

Now for the next step I think you need root access to the dev server. You basically have to run the new touch_includes.sh script, as explained in the section above. If you don't have root access ask someone on irc to do it for you when they have time.

Now go to the admin section of your dev site, say http://oxford.dev.indymedia.org.uk/oxford , and in the edit include files section you should have the two files banners.inc and style.css!

Modify accordingly your templates

The final step is to modify accordingly your templates. After having cvs update on your computer, copy the content of your /setup_uk/etc/producer/oxford/stylesheet.template file to the style.css file in the admin section, and similarly for the /set_uk/etc/producer/oxford/navigation.banners.template to the banners.inc file.

Then, edit on your computer the /setup_uk/etc/producer/oxford/navigation.banners.template file and replace everything by the following line:

<!--#include virtual="/inc/${region.filename}/banners.inc" -->

Then, modify the stylesheet.template file; replace everything by the following line:

@import url(/inc/${region.filename}/style.css);

Finally, cvs ci, ssh to the dev server, update your changes as above, regenerate stuff from the admin section, and it should work!

---

Changes:

• Main.ChrisC - 06 Jun 2004
• Main.ZaK - 08 Jun 2004
• Main.PietroFerrari - 15 Jun 2004
• Main.LeXi - 26 Nov 2004 -- add the config to conf.d
• Main.SocieteLibre - 17 May 2005 -- added the complementary section
• Main.SocieteLibre - 17 May 2005 -- added manos' hack