TODO:

NEEDS SOMEONE TO DO:
   * Phase out the clients.loudeye links in the database and on
     all IMCs (search and replace the features database, the
     article databases, and static files)

   * Need a method of notification of network wide IMC sites of issues and
   changes such as the above

jb: 
   * implement static fix for all IMCs
   * work with stefani to rsync kropotkin docs and irc configs to
   an offsite backup location
   * revise document on loudeye cleaning

matze:
   * implement no story to tell fix for all IMCs
   * work with stefani to coordinate offsite rsync backups of inglis
   databases
    * test out new postgres and upgrade procedure on a different machine

stefani:
   * work with matze to coordinate offsite rsync backups of inglis databases
   * purchase two 100meg full duplex NICs and test them for
   stallman-inglis link fix
   * work with jb to get kropotkin offsite backups rsync'd off

micah:
   * fix inglis bad disk, firewall and MTA problem
   * propose to imc-tech the purchae of two NICs to solve the
   * stallman-inglis link problem
   * work with jb, deanna and gimpboy on new mailmachine
   * email hertz@security.ddts.net to see if this person wants to
   get involved with building out new mailmachine
   * Speak to deanna about moving sarai backups to a different
   machine


gimpboy:
   * work with jb, micah and deanna on new mailmachine. 
   * Speak to deanna, or find her original post about proposed specs, so we
   can move this forward. Get specs on wiki

akb:
   * speak with shoji about getting the RAM tested and the CPU fan
   replaced on sarai


IMC-SYSADMIN meeting Jan 6th 2003 Summary

Agenda:
   imc-tech meeting overlap
   tellstories
   database backups
   postgres upgrade
   inglis disk
   stallman - inglis link
   sarai reliability / mailman upgrade
   kropotkin dependancy
   loudeye clean

* imc-tech meeting overlap summary: 

There was a meeting yesterday for imc-tech, which was odd considering
the imc-sysadmin meeting was called a week prior. The imc-sysadmin
meeting was not posted to imc-tech by mistake. We had a small summary
from akb about some of the things that were discussed there that we
related to similar agenda items for today's meeting.

We also discussed the issuse of if it makes sense to have two
meetings, one for imc-tech and one for imc-sysadmin, since there is a
lot of overlap. We decieded that there wasnt a problem with having
separate meetings, especially since some of the imc-sysadmin stuff
sometimes dominates the imc-tech meetings (imc-tech meetings have been
really long in the past due to this). Imc-sysadmin may contine to
operate more as an independant working group in this style in the
future and if so, will make a better effort at coordinating with
imc-tech's meetings.

* tellstories summary:

Occasionally you would load an article and get the error page, "No
story to tell yet", matze has developed a fix for this. JB also has a
fix that implements static files, which has been implemented on only
two IMCs, this occasionally fixes the problem. The
implementation method that was decided on was to put JBs fix on all
the IMCs and matze puts in his fix for all IMCs (note: this is a
slower method, but costs less resources).

* database backups summary:

Since the move to inglis database backups have not been made. Matze is
to work with stefani to coordinate offsite regular backups via cron,
as they were before.

* postgres upgrade

There is a newer version of postgres out, the version we are running
caused a recent crash and it is recommended for other reasons to
upgrade the software. (matze posted this link as a reference to this:
http://dbforums.com/showthread.php?threadid=3D583902). The problem is
that an upgrade requires the databases to be converted. An upgrade of
the stallman tools is in order as well. Matze will work with stefani
to build a test new postgres on another machine and test out the
upgrade procedure first.


* inglis disk

There is a firewall isse and MTA problem on inglis, coupled with a
disk that has apparantly gone bad. Fortunately we have a RAID mirror
on there which has kept the datbases safe. Micah will look at all
these problems when he is local to the machine (in approximately one
week), hopefully a full backup of the databases will have been done by
then.

* stallman - inglis link

The link between stallman and inglis is bad, causing a lot of frame
errors on the ethernet card, we have tried a lot to resolve this and
discussed what was remaining. We decided that the problem probably is
the NIC on stallman and decided to get two new NICs (matching brand),
test them and put them in the two machines. Stefani will get and test
the NICs, micah will propose to imc-tech the purchase.

* sarai reliability / mailman upgrade

Deanna had proposed and architechture and micah proposed something
similar, to build out a new box to replace sarai as our list
box. (micah posted this reference to his email to listwork:
http://lists.indymedia.org/mailman/public/listwork/2002-December/007667.html)
The idea is to build out a new machine, install the new mailman on it
and migrate the lists to it. Micah, jb, deanna and gimpboy all will work on getting this new
machine into place. 

Also discussed was the stability problems. The RAM needs to be tested
and the CPU fan possibly replaced. Shoji has arranged for others
(deanna, stefani and matze) to be able to call the colo facility to
have the machine rebooted in the future, instead of us having to
depend solely on him. akb will speak with shoji about getting the
testing done.

* kropotkin dependancies

Kropotkin was intended as a development server, with a focus on
freenet development. It is being used less and less as a dev server
and more and more for critical communications infrastructure. It is
not redundant and critical information is not backed up. jb will work
with stefani to get this stuff backed up and we will have to talk in
the future about putting in another machine, which requires a larger
discussion about sysadmin affinity group possibilities and affinity
groups that akb brought up.

* loudeye clean

jb gave a report about the mess on loudeye. We have to clean it up
because we dont have much space left. There are 543309 files on
loudeye right now, 227479 of these are unique and useful, leaving
315730 which are duplicates and need to be cleaned out. This is not a
simple issue and requires some head hitting the keyboard to figure out
all the difficult issues.

Micah will contact troy to talk to him about what the status of
paranode is, see if he can get a redirect put on loudeye to send all
requests for clients.loudeye to paranode, see if he will run a script
that we provide to remove, rename and create hardlinks hundreds of
thousands of files instead of doing it through ftp, or get shell
access (temporary), also ask about gnutella and turning on the
metafile generator on and install the real server.

We need to phase out all clients.loudeye links in the databases and on
ALL IMCs networkwide because this DNS change is not permanant.


<micah> ok, lets go!
1-7:- Topic 0(#meeting6): changed by micah6: meeting overlap
* akb/#meeting raises
<micah> go ahead akb
<akb> at the meeting yesterday we talked some about sarai
* micah raises
<akb> shoji was there and is having a couple others be able to reboot sarai by calling into the noc
<akb> also 2nd MX was discussed
<akb> it does seem inefficient to have 2 meetings w/ overlap
<akb> and i note that this meeting wasn't on imc-tech
<akb> end
<micah> i thought this meeting was specifically called to deal with sysadmin issues... but I am wondering if it makes sense to have two meetings to separate tech from sysadmin.
<micah> howevver, I didnt know about the imc-tecch meeting until after it happened
<micah> end
* jb/#meeting raises
<micah> go
* akb/#meeting raises
<jb> i proposed the idea of the meeting 
<jb> weird i didnt send the mail to imc-tech. i thought i had.
<jb> end
<micah> akb?
<akb> there seems to be some sustainability and communication issues here
<akb> i had some thoughts on affinity groups that i never sent out
* jb/#meeting raises
<akb> maybe this is too big an issue for this meeting
<akb> end
<micah> go ahead jb
<jb> yeah, i found very weird that there was a tech meeting yesterday since i thought i announced that one.
* micah raises
<jb> but since the e-mail didnt go to imc-tech that make much more sense.
<jb> while we all know there was some frictions these days
<jb> i think this is independent.
<jb> but it wont help.
<jb> done.
<micah> so do we want to try and continue to couple sysadmin issues (which tend to be more technical) along with the imc-tech meetings?
<micah> Looking at our agenda here, it is quite a bit of stuff, and I think some of the imc-tecch people would be bored stiff by some of this stuff
<akb> i don't think there's a problem w/ having seperate meetings but they should be coordinated
* jb/#meeting agrees
<akb> maybe that is more likely to happen in light of both these meetings
<micah> I agree, it doesnt make sense to talk about the same issues with both groups
<akb> but we should communicate w/ imc-tech about this meeting and acknowledge the issue
* jb/#meeting notes that usually tech meeting are very sysad oriented.
<akb> anyone else want to say anything on this?
<micah> ok, so are we essentiallz suggesting that we formalize imc-sysadmin into a working group that has separate meetings from imc-tech, but acts as a subgroup of imc-tech?
<akb> stef, matze, blicero?
<akb> gimpboy, I don't think we've met
<micah> I would like to see imc-tech take on some of the non-sysadmin issues
* matze/#meeting thinks that sysadmin/tech should be one meeting
<micah> but, if it is just going to be the same people at both...
<stefani4> some issues are inextricably linked, others not.
* jb/#meeting raises
<micah> maybe we should just continue to have sysadmin specific meetings, when the need arises
<micah> ooops
* micah points at jb
<jb> while many things are linked. it is true that tech meeting once were 6 hours long.
<jb> maybe splitting wouldnt hurt.
<jb> done
<micah> also, akbs ideas about forming specific affinitz grups would affecct this
<micah> ok, should we wrap up this topic and move on? Lets bring this to the imc-tech list, acknowledging the issue and the consideration that we may continue like this
* jb/#meeting agrees.
<micah> anyone want to say anything more about this?
<micah> anyone want to volunteer to send a quick note to imc-tech about it?
<akb> should that go w/ a meeting summary?
<micah> yeah, probably should...
<micah> ok, lets move on to the next topic
<micah> matze: do you want to talk about the tellstory stuff?
-7:- Topic 0(#meeting6): changed by micah6: tellstory 
<matze> yep
<matze> last days i wrote a script to deal with the 'no story to tell'-articles. it seems to work
<matze> fine and i wonder whether ppl are interested in implementing it.
<micah> what does it do?
<matze> find all article.*html files in local/webcast/cache that are smaller
<matze> than <x> bytes. 300 seems to be a good value for <x> for the 'no story
<matze> to tell yet' articles. check whether the file is really an article
<matze> (there is a lot of garbage in the cache folder). if it is do a refresh
<matze> of the article, if not delete the file (the file doesn't have any use,
<matze> it's an unexistant article that never will be shown and only steals
<matze> our time when we're identifying the real articles).
<matze> question is what's the best way to integrate it:
<matze> 1) one instance for all imcs (slower and lower cost) 
<matze> 2) one instance per imc (probably faster, but higher cost)question is what's the best way to integrate it:
<matze> 1) one instance for all imcs (slower and lower cost) 
<matze> 2) one instance per imc (probably faster, but higher cost)
<matze> the refresh needs its time
<matze> for 20/25 imcs
* jb/#meeting raises
<matze> end
<micah> go ahead jb, thanks for the summary matye
<jb> dunno if ppl remember of the static pages stuff
<jb> moving front.php3 active pages to static html pages.
<jb> well, i have implemented it for mayday uk and belgium so far.
* micah remembers
<jb> maybe it would do to do it also for others imc.
* micah raisses
<jb> it may solve parts of the 'no story to tell problem'
<jb> but not all cuz the active db i a bit crappy.
* matze/#meeting wonders how the probs are solved
<jb> btw i vote for 1 instance of the tell_story for all.
<jb> done
* matze micah:
<micah> it sounds like jb your fix is intended for something else, with the bonus of fixing sometimes the no storz to tell problem
<micah> so I vote that we put jbs fix into place and put matze,s number one in place
* jb/#meeting agrees
<micah> end
* jb/#meeting also agrees
* matze/#meeting also agrees
* akb/#meeting is amazed at jb and matze's ability to extend active's life
-7:- 6rabble [irc@localhost6] has joined #meeting
<micah> so am I! heh
* jb/#meeting and matze are reformists :))
<micah> ok, sounds like a victorz
<micah> speaking of which, I would like to talk about sfactive and stallman at some point
<micah> ok, anything more on this?
<micah> moving on to database backups
<matze> since we did the database migration to inglis we don't have make backups of the databases
<matze>  of the imcs on stallman. i wrote a script that does a periodical backup of the dbs. after 
<matze> a lot of headache and a semi-crash on inglis it works basically. after the problems with the
<matze> full harddisk i fear to plub it in without observation
<akb> is someone logging?
<matze> s/plub/plug it in/
* stefani/#meeting is logging
<micah> I am
* stefani/#meeting raises
* micah adds inglis disk to agenda
<matze> i ran it soemtimes, but never without being near the keyboaard
* jb/#meeting raises
<micah> matze finished?
<matze> end (more about ther problems in the link i gave before)
<micah> stefani?
* micah rasies
<stefani4> up 'til i got "employed"   db backups were rsyncd off site 
<stefani4> with the move to inglis, that terminated.    how big are the db dumps?
* matze/#meeting didn't know that. sorry
<stefani4> and how much disk space is used by the dumped files?  
<matze> +/- 1GB i think
<stefani4> so if the dumps are done individually, by imc, they could sequentially be cpied to remote?
* matze/#meeting connecting to inglis
<matze> yes
<stefani> one GB is certaily within the range of space available on several remote machines.
<matze> mayday is about 100mb i think
* stefani/#meeting ends
<micah> jb?
<jb> matze you have been able to do only 1 mayday backup, no?
<matze> yeeeeeeeeees :)
<jb> :)
<matze> i did the first
<jb> weird
<matze> and hope we can do them reguallary now
<jb> one question
<jb> i wonder if it is possible to do backups a la
<jb> tar zcvf 
<jb> instead of using the pgsql mechanism.
<jb> done
<micah> no, it is not a good idea to do it outside the db mechanism
<micah> because zou will get ccorrupt databases, not good for backups
* matze/#meeting sees that the dumps only ocupy 351M
<micah> zou can do that, if you shut down the database first, but that is too disruptive
<stefani> but you can gz the dump.
<micah> so I think we should have regular bacckups, rsync'd offsite
* matze/#meeting the dumps are already gz
* jb/#meeting wonders if ppl know of bandwidthcoop troubles.
* stefani/#meeting agrees with micah, as this was done previously, as an added safeguard.
<micah> so, stefani, can you work with matze to get offsite rsyncs of backups going?
* stefani/#meeting nods
<matze> fine ;-)
<micah> cool, that would be great
* jb/#meeting applauses
<stefani> backups   is my middle name
<micah> we are doing great here. :)
<micah> ok, we move on then?
<micah> matze?
<matze> it's recomendable to update to postgres 7.3
* micah raises
<matze> our semi crash was related with a bug in the prior versions
<matze> http://dbforums.com/showthread.php?threadid=3D583902
<matze> prob is that the databases need to be converted
* micah cringes
<matze> this means we have to trust our backups ...
<matze> end
<gimpboy4> it needs to be dumped and reinserted i believe
* jb/#meeting raises
<micah> ok, so can we try doing it like this
<stefani> but we could beuild a test pg on another machien  and test the thing out first.
<micah> test our backups by trying to restore to them
<micah> and then doing the upgrade?
<micah> or, do as stefani suggests
<matze> stefs proposal sounds good to me
<micah> I wanted to bring up another thing related to this, once we have finished with this subject
<micah> kropotkin?
<jb> btw we have postgres running on stallman these days.
<micah> yea, what I was going to also say is that we need to upgrade the pgtools on stallman, so remote database acccess to inglis can work
<matze> yes, although i'll ask for help when it's getting concrete
<micah> end
<jb> ok
<micah> jb, did you have something else to say?
<jb> no, wanted to talk about pgsql client part on stallman.
<jb> that's done.
<micah> ok, cool... moving on then
<micah> does anyone mind if I insert inglis' disk in here
<micah> since we are talking about inglis?
<jb> go on
<akb> ah, that would be good
-7:- Topic 0(#meeting6): changed by micah6: inglis disk
<micah> ok, so... I am going to fix this problem
<micah> but, not until I get back to the states, where I am near thebox
<micah> I am too nervous about messing with it from argentina
<micah> there is a chance it isnt a bad disk, but just needs a reboot and a raid resync
<micah> there is also a weird firewall issue and mail problem on there 
<micah> you cannt get traffic back from the internet that was initated from the machine and the MTA was screwed up
<micah> both of which I will fix when I return
<micah> I will be home mid january
<micah> end
<akb> would a reboot of inglis cause a problem do you think?
<micah> I am hesitant to reboot, although I am reasonably certain it would reboot fine, I just dont want to take the chance
<micah> a reboot might fix it
<akb> i meant if we tried to fix the nic problem
* jb/#meeting notes that before rebooting we can do a tar zcvf of the db just in case.
<akb> maybe we should hold off on that?
<micah> waiting about a week should be ok in mz opinion
<micah> akb: I dont understand
-7:- Topic 0(#meeting6): changed by micah6: stallman - inglis link, inglis disk
<akb> i meant if we were going to put in a different nic we would have to reboot which could cause a problem
<akb> but if you're going to be back soon then we should wait on that too
* matze/#meeting agrees 
<micah> right, however... I dont think it is the nic, because I switched it before, remember?
<akb> you switched nics?
<micah> there are two nics in inglis
<micah> yeap
<akb> huh
<micah> yea
<stefani> two?
<akb> same type of card?
<micah> me nods, one in the scsi ethernet combo card, and then one other on built into the motherboard
<micah> no
<matze> did you exchange the nic of stallman too?
<micah> no, I dont think so
<matze> the stallman link is running half-duplex
<matze> and doesn't respond to the mii-commands
<micah> yea, I tried to force that to full, but it wouldnt work
<micah> exactly my experience
<matze> i think we have to exchange this nic
<micah> the stallman one
<matze> yep
<micah> I agree
<matze> with a high quality one ;-)
<akb> i thought it was an eepro
* micah notes that the stallman second nic is a repc nic
<micah> turtle was built from repc.... heh
<matze> akb: eepro is eth0
<micah> well, i think it is an imitation ne2000?
<akb> ah
<micah> i cant remember
<matze> Digital Equipment Corporation DECchip 21140 [FasterNet]
<micah> so, when I get back I will switch stallmans nics, reboot inglis and look at the drive
<matze> plis
<micah> can do that all in person
<stefani> eepro100 or 3c905tx 
<micah> it is a tulip
<matze> right
<micah> I could probablz take the extra nic from inglis and put it in stallman
<micah> but mazbe it would be better to just get a different one
<micah> reduce the insanitz
* stefani/#meeting proposes new nics. matched
<micah> I feel like II am talking in hax0r speak
* micah agrees with stefani
* stefani/#meeting proposes matched/tested in other boxes first.
<micah> problem is testing two nics is not so easy
<micah> I onlz have one computer
<micah> :)
<micah> ok, I will propose to imc-tecch that we get mmonez to buy these
<micah> moving on....
<micah> sarai
-7:- Topic 0(#meeting6): changed by micah6: sarai reliability / mailman upgrade
* jb/#meeting raises
<micah> go jb
* micah raises
<jb> deanna had proposed an architecture for adding a box 
<jb> to handle spamassassin and things like this.
<jb> unfortunately i cant find the reference.
* akb/#meeting raises
<jb> the point is that she is the mail/mailman/spamasssassin wizard
<jb> and we're a bit lost at listwork without her.
<jb> done
<micah> I was suggesting in an email to listwork something similar
<micah> that we get to work at replacing sarai, and in the process, while the replacement is almost readz we put the new mailman on the n ew machine, migrating everzthing
<micah> http://lists.indymedia.org/mailman/public/listwork/2002-December/007667.html
<micah> I dont think this is something that deanna can or wants to do all on her own
<micah> the migration to the new mailman will not be simple, although it will be simplified if we have a separate box to do it on
<micah> I am willing to help with this
<micah> end
<akb> were there configuration changes to sarai after deanna said there was a big problem?  load had gone down before the new RAM went in
<akb> the RAM didn't go in until fairly recently and things didn't look as critical in terms of queue length
<akb> but i hadn't kept such a close eye on things
<akb> the other thing we should talk about is sarai's reliability
<micah> the onlz thing is the CPU fan that stefani noticed, I dont know of anzthing else
<akb> we dealt w/ the rebooting issue yesterday but not the cpu fan / ram testing
<micah> akb, what would zou like to talk about regarding sarai's reliability?
<micah> what was the rebooting issue?
<akb> shoji is arranging to have other people be able to call the colo to get sarai rebooted
<akb> matze, deanna, stefani i believe
<micah> ah, excellent
<micah> so the new ram is in there?
<micah> and it wasnt tested before it went in, right?
<akb> as far as what to talk about, i don't think we've heard from shoji about what how to troubleshoot either the ram or cpu fan
shoji: No such nick/channel
<akb> the crashing was fairly correlated w/ the new memory going in
<micah> has anzqadasdasd
<micah> sorrz
<micah> has anyone sent him email or tried to talk to him about it?
<akb> he was cc'd on some of the discussions, i guess we should be more direct.  i'm sorry it wasn't brought up yesterday
<micah> is the new ram in there still?
<akb> yes
<micah> ok, so I suggest we call or email shoji directly to talk to him about it
<akb> i owe him a conversation anyway, i'll do it
<micah> ok, cool
<micah> I cant find deanna\xDFs message either, maybe it was on the wiki?
<micah> I cant find deanna\xDFs message either, maybe it was on the wiki?
<micah> weird
<micah> I cant find deannas message either, maybe it was on the wiki?
* akb/#meeting chuckles
<jb> nope, i think it was an e-mail-
<micah> dang apostrophz
<micah> ok, I will work on talking to her about it and getting the balll rolling on it
<micah> we will probablz need help when it happens, so if anyone is interested.... :)
<jb> i am.
<micah> ok, next topic, unless anzone has anything more to say
<akb> maybe we should specifically recruit someone new?
<micah> I like that idea
<jb> meybe we should clone deanna.
<akb> there have been a couple new people that seem skilled and interested
<matze> ;-)
<micah> lets talk to the alien cloners in quebec
* akb/#meeting hints gimpboy
* micah looks at gimpboz
<gimpboy4> i just got back what did i miss
<micah> you were just volunteered!
<micah> ;)
<gimpboy4> for what?
<jb> congrats.
<gimpboy4> i make great cornbread
<micah> we were wondering if zou were interested in getting zour feet wet by jumping in on pushing forward the new listmachine
* blicero/#meeting congratulates gimpboy 
<micah> hahah
<micah> replacement for sarai, building it out, installing new mailman and migrating lists to it
<micah> not all by yourself of course
<micah> haha
<blicero4>       _             _      _             _ 
<blicero4>   ___| | __ _ _ __ | | ___| | __ _ _ __ | |
<blicero4>  / __| |/ _` | '_ \| |/ __| |/ _` | '_ \| |
<blicero4> | (__| | (_| | |_) |_| (__| | (_| | |_) |_|
<blicero4>  \___|_|\__,_| .__/(_)\___|_|\__,_| .__/(_)
<blicero4>              |_|                  |_|      
<gimpboy4> you want me to put it together or configure it?
* micah kicks blicero
* blicero/#meeting plauds to gimpboy braveness
* stefani/#meeting thinks we oughta spec out the hw first
<gimpboy4> i'm real good at being told what to do.
<micah> gimpboy: well deanna, myelf and jb all are probably going to work on it too, but were wondering if you wanted to get involved in it too
<gimpboy4> sure
<akb> hertz@securify.ddts.net also seemed interested i believe
* matze/#meeting can do certain tasks too
<akb> has sent a couple messages to sysadmin looking to get involved
<micah> gimpboy: cool... maybe you could start by tracking deanna down and finding the message she sent about specs? I agree with stefani that we need to spec out the machine first
<micah> akb: ok, lets try and get that person involved too
<gimpboy4> point me to the list and i'll poke around.
<micah> gimpboy: I looked in the list archives for listwork up until oct but didnt see the message from deanna... maybe contacting her directly?
<micah> http://lists.indymedia.org/mailman/listinfo/listwork
<gimpboy4> is it in imc-tech?
<gimpboy4> http://lists.indymedia.org/mailman/public/imc-tech/
<micah> maybe... imc-tech, listwork, or imc-szsadmin
<akb> maybe we should move on?  just thinking about matze and the time
<micah> sysadmin
* micah nods
<matze> gimpboy: deanna@freeshell.org
<micah> gimpboy: just keep us in the loop and we will collaborate to getting this thing moving
<gimpboy4> ok.
<micah> probably should get specs up on the wiki
<micah> ok, moving on
-7:- Topic 0(#meeting6): changed by micah6: kropotkin dependancz
<micah> I put this on there... erp
<micah> we have talked about this before... kropotkin was intended to be a development server, but now our wiki, irc and todo stuff lives on there
<micah> and there are nobackups, no raid or anything 
<micah> which makes me nervous
<akb> it is the backups for sarai
<micah> that too
* stefani/#meeting hands micah a trnquilizer
* micah takes tranq
* stefani/#meeting says tranq0you
<micah> so i dont know, it is sort of a big issue... how to deal with this
<micah> but, it is also a big issue if the disk should crash and we have no backup of the wiki
<micah> which people are reallz starting to work on and depend on as a communications tool
* stefani/#meeting thinks it sounds like time for a cron script and rsync
* jb/#meeting is ok to find a rsync tutorial and help stf
<micah> yea, that would be a temporary fix, but reallz... the point is that the machine isnt built for stability and was put online with a different intention and we are sort of violating that intention
<matze> are you thinking to setupo a new box?
<micah> it was putonline to be a dev server and a freenet node
<micah> and the person who put it online thought it was onlz going to be a freenet node (and used to develop for freenet and IMC), when we said it was to be a general dev node he was a little surprised
<micah> and now it is a lot more
<micah> I dont know... that would be one solution I suppose, but maybe we can just move docs to another machine
<micah> but I think backing it up via rsync is best for now
<micah> end
<micah> well, not end
* jb/#meeting notes that the general tendency is that we need more boxes.
<micah> I think moving sarai backups off of kropotkin would be good, and if backups happen then we should backup irc configs, and docs
* jb/#meeting raises.
<micah> end
<micah> go jb
* matze/#meeting remebers that shiji said that philly just bought two new boxess and would like to hos indy stuff
<matze> s/shiji/shoji/
<jb> i note that that the general tendency is that we need more boxes.
<jb> sorry, a bit off topic.
* micah hears an echo
<jb> i think we have to consider what akb wrote about comunity sysad
<jb> not sure comunity is the word.
<jb> before expanding.
<micah> judi is available for things according to rabble, but it needs some work... also, the portland machine has been offerered
<jb> done.
<micah> I agree
<jb> i propose we planify a bit.
<micah> planify?
<jb> clarify what boxen can be made available, see who can be in charge, etc.
<jb> not now, i mean.
<micah> is that a form of plan?
<jb> yeah :)
<micah> I agree... for now, jb can you work with stefani to get an rsync setup of that stuff
<jb> ok
<micah> haha
<micah> ok, next
* jb/#meeting would like to tell the world that micah is laugthing at him for his english.
<micah> last!
* micah laughs at himself for his english dependancz
-7:- Topic 0(#meeting6): changed by micah6: loudeye clean
* blicero/#meeting pukes
* jb/#meeting raises
<micah> go jb
<jb> so
<jb> i sent an e-mail to sysad and tech iirc
<jb> we have 500,000 files on loudeye
<micah> woops, wrong window
-7:- 6pietro_ [~pietro@host21.colband.com.br6] has joined #meeting
<jb> about 60,000 are useless or unidentified
<jb> about 200,000 files are really usefull
<jb> the rest are duplicates :)
<jb> now i have an account on jpl box and can do some stats.
<jb> the plan is to get rid of duplicates 
<akb> half are duplicates!
<jb> no
<micah> more than half
<jb> a good file has an average of 1.5 duplicate
<jb> sorry that was 560,000 in the beginning
<micah> 560,000 useless
<jb> since we dont have shell access to loudeye it is a bit tricky to do this properly
<akb> how about coordinating with troy?
<jb> would be good.
<akb> he would probably willing to do whatever to get things cleaned up
<jb> :)
<micah> I can easily write an expect script to do it too, but I think if we had a list of files to remove and gave it to troy it would be less resource intensive for him to do it
<jb> getting a shell access or having him running a code would be nice.
* akb/#meeting notes one of the topics yesterday was what exactly is the relationship w/ troy and what is paranode
<jb> the alternative is to do 200,000 or 500,000 "mv" by ftp.
<micah> akb: was that resolved
<micah> ?
<jb> and 300,000 "rm"
<akb> there were some questions that we wanted to ask i believe
<micah> ok, I will contact troy about that, I would like to touch base with him about where things are at anywas
<akb> i think there was some unease that we didn't hear anything for a while
<micah> akb: if you can remember the questions, I can either answer them, or get troy to
<akb> and then dns changed w/o warning
<micah> yes, there is a problem with this
<akb> two things:
<micah> becyuase I think some files reference the clients.loudeye link
<micah> jb: was there more to discuss about the cleanup? like if we were going to crawl the databasse to do URL renamings, etc?
<akb> should we move files for imcs that aren't using loudeye / paranode anymore off if space is an issue (italy, argentina)
<jb> micah yes
<akb> i asked troy about running gnutella on the server, he seemed open to it
<jb> we need to find a way so that the http queries find their way to the files.
<jb> we have various possiblities:
<micah> akb: where are these other sites hosting their stuff? gnutella?
<jb> * use apache redirects for that
<micah> ok, lets focus on jbs issue first
<jb> * rename files in the postgres database
<jb> that's all i think so far.
<jb> the point is there is such a quantity of filenames
<micah> and we dont know which file each story points to
<jb> that it is a bit hard to tell whats better
<micah> so it seems we need to:
<jb> so far most of the things i try to do on my box lead me to 'out of memory' or similar probs.
<micah> 1. identify duplicates (done)
<micah> 2. idenify which files are actually referenced, and which arent
<jb> sorry micah, go on.
<micah> 3. remove the unreferenced files
<micah> 4. rename actual files to MD5 sums
<micah> 5. rename in database
<micah> anything else?
<akb> 3 may not actually be safe
<jb> err
<micah> akb: why?
<akb> i bet there are imcs that have files there that aren't in postgres
<micah> hmm, NYC for example
<micah> yyes, but if we have:
<jb> i dont know if 3 is actually doable
<akb> jb: how were you thinking of using apache redirects?
<micah> MD5sum matching two files
<jb> (or a script)
<micah> and the database has no reference for either of them, then we dont delete either
<jb> akb: the active installs are setup NOT to mirror, on stallman
<micah> akb: you should be here in argentina :)
<akb> micah: that's mean :(
<jb> so that the queries are to the imc /uploads/metafiles directories
<akb> jb ?
<micah> akb: aww, I was meaning it ass a compliment, it would be cool
* jb/#meeting restarting :)
<micah> err, not ass
<akb> heh
<micah> jb: are these files non loudeye files? or jjust biyare?
<micah> bizare
<jb> 1: there is no mirroring parameter on stallman'active install, as a consequence, the multimedia queries are done to http://www.indymedia/local/webcast/uploads/something.gif
<micah> for certain sites?
<jb> so we can use apache redirect to send this to images.indymedia/mayday/something.gif
<jb> for all sites i think
<jb> this is what we do now
<jb> we can plug in the redirect a query to a dbm file or something so that the redirect sends you to
<jb> images.indymedia/mayday/md5-hashed-name
<jb> this is possible.
<jb> i dont know if it is the best thing to do.
<akb> what about external links to images.indymedia.org/....
<jb> ej
<akb> renaming on paranode will break things w/o some mechanism there
<akb> not all imcs that use paranode are on stallman (chiapas, philly, dc, etc)
<akb> its complicated
<jb> yes
<jb> maybe there is something we can do
<jb> there is an apache option that let you put an http response in the file
* stefani/#meeting points out that nearly 120 minutes have elapsed, and will have to depart withing 20 minutes.
<jb> if we replace all files by a redirect response we are safe
<jb> but this involve config changes on loudeye.
<jb> other thing is we can clean only for stallman imc.
<jb> they are the more bogus actually.
<jb> (when it comes to mirroring).
<jb> one last thing:
<jb> about renaming what is in the database and cleaning unreferenced files.
<jb> it involves 250,000,000,000 queries :)
<jb> thats 500,000^2
<jb> i think.
<akb> gah
<jb> done, cheech.
<micah> yeah, I was going to suggest using the new postgres install on another machine that matze and stefani are going to do for the upgrade to do some of this
<micah> otherwise, I dont think inglis can handle that
<jb> :)
<jb> that's worst case, maybe we can find a way.
<micah> unlss we turned off access
<micah> so, I am a little confused now
<stefani> not likely on a 'live' srever
<micah> what needs to be done, decided and resolved?
<jb> i am confused also.
<akb> jb: maybe you can refine the document you posted w/ this discussion?
<jb> maybe shell access ...
<jb> ok.
<akb> and contact troy to see about coordinating it
<akb> one more thing b4 we close this issue
<micah> it seems like we need to define a plan of action
<akb> oh sorry finish this, mine's not directly on cleanup
<micah> or acttually, what we need to do, is decide on the best way to deal with this?
<micah> or is the issue to resolve the coplicated problems, then find a best course of action?
<micah> would it just be easier to remove everything and start over?
<micah> so I suggest, for simplicity sake, we work on the stallman mirror first
<micah> it has the majority of the problems, and removes some of the complexities
<jb> i dont understand the stallman mirror
<micah> I mean, the sites hosted on stallman
<jb> ah yeah
<micah> as you said:
<micah> <jb> other thing is we can clean only for stallman imc.
<micah> <jb> they are the more bogus actually.
<micah> <jb> (when it comes to mirroring).
<jb> maybe we need to talk about pushtoeye.pl ?
<matze> bona nit ppl
<jb> and possible replacements.
<jb> noche matze
* stefani/#meeting has to go too.
<akb> hmm, should we stop?
<micah> so i sounds like doing a redirect on loudeye wont work because of the sites outside of stallman
* stefani/#meeting bows out
<jb> chau stef
-7:- stefani 0[irc@localhost0] has left #meeting
<micah> ok, sounds like people are dropping off... but jb and akb are you still willing to work on this problem?
<micah> me is
<micah> but akb you had something else to say
<akb> i can keep going
<jb> i can also
* micah needs to type up the notes, that wont be fun on this computer
<jb> i can let you mine
<akb> i wanted to note that we need the metafile generator turned on
<micah> what is a metfile generator?
<akb> troy said he would do that and install a real server
* blicero/#meeting so micah can edit the notes on a wonderful italian keyboard
<akb> clients.loudeye.com/seattle/foo.ram
<akb> was generated by a script
<akb> all the links pointing to these are broken atm
<micah> ooooh
<micah> ok, so I should ask troy about that too?
<akb> also we should phase out clients.loudey.com links in general
<micah> yeah, that may require some database work
<micah> because I think there are links on a lot of articles (and outside servers too)
<akb> well we'll have to search and replace the features database, the article databases, and static files
<akb> and probably have troy configure a virtual host for clients.loudeye ...
<akb> that sends a redirect and whatever the http code is for permanent relocation
<akb> and maybe give us the referrer logs 
<akb> sigh
<akb> did that make sense?
<akb> hello?
<jb> sorrz
<pietro_4> akb, that made sense :)
<jb> we were switching kezboards
<akb> heh
<micah> yeah, that makes sense, except that isnt clients.loudeye ued by other loudeye clients
<micah> so doing a redirect would have to be specific for us
<akb> i believe it is not
<micah> oh?
<akb> clients.loudeye.com is now a cname for stream.paranode.com
<micah> interesting
<micah> and paranode is his box under his bed?
<akb> rabble said something that made me believe that was the case
<micah> well, unrelated
<micah> ok, hrm
<micah> if he does a redirect, do we need to do the search and replace on the database?
<akb> well, we should
<micah> for cleanliness sake
<akb> he said the loudeye dns will probably not be forever
<micah> I just dont see making such a radical change in the database as being something very easily accomplished with inglis heaving under its load
<akb> and if we have to make things even more complicated down the road doing search and replace now will make that easier :)
<micah> heh, true
<micah> also, how do we notify other IMCs who dont use stallman?
<akb> we can set low priorities on the queries maybe?
<micah> yeah, I dont know how to do that
<akb> that's a huge communication infrastructure topic i think
<micah> akb: yeah, I agree... :p
<micah> ok, based on this discussion are you volunteering to do the database changes? ;)
<akb> i don't think chiapas has been plugged into the loudeye changes for instance
<akb> ug
<micah> definately need to work on a method of notification
<akb> i guess so
<micah> maybe matze would be into helping, he is turning into our regular DBA
<micah> :)
<akb> i am worried about not being able to follow through
<akb> too many commitments
<micah> no kidding...
<akb> i've been letting some people down
<jb> mazbe we can trz to see with troz to start with
<akb> the search and replace isn't critical at the moment
<akb> cleaning up is
<jb> see if we can have a shell or a code run on loudeze.
<micah> ok, that is good for you to be aware of, lets not pile this on you then
<akb> as is communicating w/ troy and each other and making current links work (a la meta file generator)
<micah> jb: but having a shell or code on loudeye wont fix the URLs in the article databases
<micah> ok, i will take care of talking with troy... i will put the db link changes on the todo list in the summary of these notes as something that needs someone to commit to doing it
<jb> mmm, if we are able to do on loudeze the same scripts we have on stallman, we have no immediate need to clean the database.
<micah> jb: which scripts?
<jb> ah, sorrz, i am off  topic.
<micah> ah, you were talking about the file clean, not the cleints.loudeye change in the database
<micah> i think we are done with that topic now, right akb?
<jb> zes.
<akb> yeah
<akb> oh i could mention gnutella
<micah> yea, whats that about?
<akb> since its almost completely frivolous
<micah> good to distract ourselves from the difficult loudeye clean
<akb> i was thinking of inserting dc imc content into the bitzi catalog
<jb> its the same as kayaa no?
<micah> kazaa
<micah> yeah, it is similar
<jb> kazza
<akb> kazaa is closed and commercial
<micah> but what would having it on loudeye be for?
<jb> fcking kezboard :)
<akb> gnutella has an open spec
<akb> well the bitzi catalog helps be able to find content on p2p
networks using digital signatures
<jb> blicero: first intent of traffic shaping for the fscking neighbor
<akb> thus it would potentially expose a larger audience (the p2p
audience) to imc content
<akb> and it makes noninfringing use of p2p
<akb> which will become more important as the Creative Commons
licenses gain prominence and the RIAA and MPAA crack down on pirated
content 
<akb> or maybe its a waste of time
<jb> i agree
<akb> its a waste of time?
<jb> no :9
<jb> definitelz, i think it is important to go frreenet and p2p
<micah> yeah, i think it would be good to... he has already said he
would turn it on?
<akb> he was into the idea
<akb> there's no real good headless implementation to run
<akb> that can run as a daemon
<micah> hmm, how should I bring it up with him?
<akb> i'm not sure
<akb> i would say it has low priority
* jb/#meeting has to go
<jb> see za
<akb> maybe just mention the idea
<akb> bye jb, thanks
<akb> damn missed 'em
<akb> well, i guess we're done
<akb> did we cover everything?
<micah> he is here
<micah> sorry, we are sharing a computer now
<akb> ah
<akb> cozy and efficient
<micah> haha
<micah> the cleaning loudeye problem is still outstanding
<akb> what else besides jb revising his document and coordinating w/
troy about shell/script access
<micah> well, we could go into trying to solve the problems (or
identifying what the problems are)
<micah> but we could just close things up for now
<micah> :)
<micah> what do you think akb?
<akb> i'm kinda tired, we're approaching hour 3
<micah> and anyone else not speaking... :)
<micah> we got a lot finished in the time
<akb> yes
<akb> and it felt good to have a meeting again
<micah> yeah, it does definately
<micah> too much up in the air
<micah> I will get the notes, summary and what we all said to do out soon
<akb> cool