You are here: Foswiki>Global Web>WebHomeWG>DeadImc (26 Sep 2009, BouD)Edit Attach

DeadImc

Table of content :

An attempt was made to start a proposal on the global level to deal with the dead links in the cities.inc list (http://www.indymedia.org/cities.inc) which is automatically generated from the imc contacts database. As many as 1 in 6 of the links led to what seemed to be non existing or closed websites. The resulting discussion focused both on the difference between a website and an indymedia collective and on errors in the list of apparently dead links sent with the proposal. There is currently some confusion regarding the difference between a defunct indymedia website and a dissolved or malfunctioning collective and processes for dealing with these: This page is a place for thoughts on this issue to be gathered with a view to perhaps forming a working group or some other process by which to deal with the mounting problem of the desire by some imc volunteers to have an elegant software presentation of the nature of the imc network and their impatience in reading mailman archives.

Some response from global lists

John

"it's definitely something that needs to be done....i think a proposal
to create an inactive/archive category would find widespread support,
and that'd be something i'd be willing to help draft....maybe put
something on docs.indymedia.org, include the list of imc's which are
thought to be inactive..." - john

Boud

If you want to make a proposal for declaring certain collectives as
no longer existing as *collectives of activists doing independent media*,
then please think up a reasonable process, propose it first to your local
collective, and after their corrections and if they accept it, then propose
it up to the "network" on imc-process.  Even better (see my comments a 
few sentences below), try getting some consensus at your own regional level.

Since this is not yet at the proposal stage, here are my suggestions
(not of my collective):

* First regionalise. There's no way we're ever going to become a global
network (rather than semi-global) if good english speakers with good 
internet access can dominate and e.g. easily throw out other collectives
from the network.

* Secondly, regionalise: Divide your specific suggestions into regions
and contact the regional mailing lists.  E.g. try sorting out the north
america based imc's in your own region before trying to claim that collectives
in other regions no longer exist.

* Three: having a holding page telling people to get reorganised if
they want open publishing turned back seems to me like a Good
Thing. That doesn't mean that the collective still exists, but shows
that it did exist and destroying a static site with helpful hints on
getting reorganised seems quite unnecessarily destructive to me.

* Four: remove the duplicate entries from your list.

* Five: with 150 or so local collectives, if each group has a week
downtime of their website per year, that makes typically three
websites down at any one time. One week's downtime is hardly a reason
for removing a collective from the indymedia network.

Proposal to UK

Hi all,

I've been trying to initiate a proposal at the global level to purge the
dead sites out of the IMC list that appears on every indymedia site.

Below is a list of 23 sites which simply do not exist at all at the time
of writing and there are many others which are broken, hacked or closed.

...
snip
...

It is possible that some of these are having temporary problems and will
return but a lot of them have been gone for a long long time.

There also are others which may be dead or may just be having technical
issues...

http://www.detriot.org/ - hacked?
http://mumbai.indymedia.org/ - hacked?
tnimc.org/ - forbidden due to permissions problem, is there a site?

Additionally there are one's we know are having technical difficulties and
know will be back...

radio.indymedia.org - server problems but coming back
kenya.indymedia.org - off but apparently coming back really soon

There are also 'frozen' projects which still have websites or holding
pages and it can be assumed that some of those will come back at some
point.

www.stlimc.org - no editorial features, collective reorganizing
hamilton.indymedia.org - closed to publishing but perhaps to return
melbourne.indymedia.org - publishing suspended, but perhaps to return
bc.indymedia.org - off for now, may relaunch
victoria.indymedia.org - off for now, may relaunch
beirut.indymedia.org - collective dissolved a few months ago

There are almost certainly others which are working sites but with no
active collective anymore.

Like in the UK network with the issue of West Country indymedia, there
appears to be no gloabl process in place to deal with dead sites or non
existing collectives.

It took several years in the UK to finally turn off Westcountry even
though there had been no collective for over three years and a closing
note had been placed on the site. It seems that a proposal to global
process will fair no better. Therefore I would like to propose that the UK
takes a unilateral approach with a view to pushing forward discussion and
eventual action at the global level.



My proposal is therefore that the UK site make use of a non standard
modified cities.inc list rather than the out-of-date list generated by the
IMC contacts database.

This new list could be manual generated or it could use the generic auto
generated cities.inc list but automatically filter out bad links based on
link validation tests. Better still would be an even smarter approach that
would take into consideration when a dead link was last seen working and
leave it in the list if it was up a few days before.

The smart approach would result in a list that still contained links to
sites undergoing short term technical difficulties and it would also
automatically reintroduce sites which were relaunched after long periods
of downtime. Obviously the automatic approach could not deal with sites
which have been closed but leave some kind of holding page but I don't
think this is so much of an issue.

Why do I think this is important and worth bothering with? I think it
gives a really bad impression that so much of the global network consists
of dead links and that it does us no favors to hang onto the idea that
there are 180  plus indymedia sites when there are not. More sites will go
- only last week melborne indymedia closed down. If Indymedia is
contracting, surely it is better that we don't hide from the fact till
half the network is gone. Clearing out the dead wood in the cities.inc
list is a way of coming to terms with the shrinking of the indymedia
network and hopefully laying the ground work for a healthier future.

I really feel that beyond the simple issue of whether to list dead
websites in the cities list, the issue of removing dead IMC collectives
from the IMC network is one that must be dealt with on a global level. It
appears there needs to be an dead-IMC process just like there is a new-IMC
process. Not dealing with this now as the network contracts is just
leaving us open for bigger problems in the future as more unmaintained
sites become spam holes and legal liabilities or get taken over by new
unaffiliated collectives and become political liabilities.

ben

Follow on taken from IRC #UK

[12:06] <chrisc> ben: a script could be written to do the cities list that checked for the http status code of each site
[12:14] <ben> chrisc yes once a day and compared with previous status
[12:14] <chrisc> right
[12:14] <ben> then you just need thresholds for when a site is considered dead
[12:14] * chrisc nods
[12:15] <ben> down for a week perhaps mean exclude it
[12:15] <ben> could even apply different css class to each site dependent on it's status
[12:15] <chrisc> or, to make it simpler, it could only list sites that have a 200 response and not do any complicated history / threshold stuff
[12:16] <ben> that would certainly be simpler but it would mean that temporary dead sites would be hidden from the list - not sure is that is a problem
[12:16] <chrisc> yep, would be hidden for 24 hours at most
[12:17] <ben> it might have just been busy and timed out, would be a shame to loss it from the list in such a case perhaps
[12:17] <chrisc> yeah, true
[12:18] <ben> whereas, if the script has checked the dy before and it was down and it's still down now... that's a fair sign that there is a problem
[12:18] * chrisc nods
[12:19] <ben> I think even with my non programming skills I could take the phonifier script and make it do the required task
[12:20] <ben> don't know how to save a new doc though
[12:20] <ben> could probably sus it out
[12:21] <chrisc> we have a cron job running wget to get the cities list every night
[12:21] <ben> so... take cities.inc, feed it into a link validation program, regex out those returning  500, 503 status etc. reformat page so it's the same as cities.inc , job done

More IRC chat on this topic

toya: i am changing that wiki cuz india aint dead u checked it by hand
[5:02pm] toya: at the same day the server was down
[5:02pm] toya: i just sent an email
[5:02pm] toya: that if we want to do this work we cant just go using a script or clicking on the sites
[5:03pm] toya: we need to contact the collectives and / or
[5:03pm] toya: the ppl whho are hosting those sites
[5:03pm] toya: this is a good work to be done but cant be that way
[5:03pm] pabs: I think what this reveals is that a lot of sites have tech problems
[5:05pm] toya: india and chilesur were 'dead' cuz the server was down that day you clicked on it..thats all ..is froozen by a request of the collectives who arent active now but want the content to be avaliable for ppl to access it and we (imcbrasiltech) who host it has to respect that
[5:05pm] toya: we cant just remove them
[5:06pm] toya: we are going to migrate them to a new and better server pretty soon
[5:06pm] toya: all this work is happenning and i bet with many other sites on that list ppl has some history to tell about what is the problem and what is the solution they are working with
[5:07pm] johm: we really do need to come up with a more formal approach to archiving old sites....
[5:08pm] pabs: one problem is all these dynamic codebases
[5:08pm] gdm: johm: yeah, and some definition of what an "old" or "dead" site is
[5:09pm] toya: what bout froozen ones?
[5:09pm] toya: is that old?
[5:09pm] toya: is that dead?
[5:10pm] johm: really, i'd love to see an imc crawler that just archives sites continously, so that when a site has tech problems, or the collective behind it falls apart, the content is still there
[5:10pm] toya: johm: has u seem the archive proposal that is beeing building?
[5:11pm] johm: i talked to people about it, but i haven't seen any proposal yet
[5:11pm] toya: talking with micah and other ppl i started this
[5:11pm] toya: http://docs.indymedia.org/view/Global/ImcArchives
[5:11pm] toya: i never sent cuz it needs to be finish
[5:11pm] toya: the proposal needs to be finish
[5:12pm] Kameron: would be good to get statistics like what percentage of an archived sites stories were hidden
[5:12pm] johm: it would be nice if the contact db was used to generate the list of sites to be archived
[5:13pm] Kameron: is there an 'archived' field there anywhere? or something that could indicate as such
[5:14pm] johm: ah, that's a problem - how do you handle page updates effectively?  i guess you just have to do it like archive.org does, and keep multiple versions by date....but then what happens to removed content on the local site which has been archived?  i guess, just like archive.org copies, it just stays up
[5:15pm] johm: Kameron: there's a status field, right now dead imcs get marked "inactive"
[5:16pm] Kameron: this archive is another place we are going to have to look to remove stuff from, in case it sucks stuff off the local imcs before a hide can be done
[5:17pm] \x95 ben__ ask's it anyone minds if i log this an post it on docs?
[5:17pm] johm: ben__ : Please please do!
[5:17pm] toya: yah
[5:18pm] johm: i'd say, there should be archive.indymedia.org
[5:18pm] toya: johm: this is what the proposal i was talking about
[5:18pm] toya: talking on #sfkids
[5:19pm] toya: tech indymedia has the money for a machine
[5:19pm] toya: and global to keep the connection/host
[5:19pm] toya: for it
[5:19pm] johm: and if a site is in the contact db with status either live or inactive, you should be able to find it at archive.indymedia.org/$host, where $host is also what's set in the contact db
[5:20pm] toya: we should also have places in there for our own things - like docs, internal imc stuff
[5:20pm] johm: toya: i'd love to help think this through and work on it.
[5:20pm] johm: i wonder how much diskspace you'd need....
[5:20pm] toya: tons
[5:20pm] toya:
[5:21pm] toya: is good to know you would like to help with it
[5:21pm] johm: Kameron: it's a tough problem, the hidden content issue
[5:23pm] pabs: LETS GET A PENTABOX!
[5:24pm] pabs: woops, capslock
[5:24pm] Zapata: YEAH, LET'S DO THAT
[5:24pm] Zapata: BUT WHAT IS A PENTABOX?
[5:25pm] pabs: I mean petabox: http://www.archive.org/web/petabox.php
[5:25pm] Zapata: right
[5:25pm] Zapata: never heard of that either though
[5:25pm] micah: ssd
[5:34pm] ben__: archive.indy relates a lot perhaps to search.indy

Proposal draft 2009

Indymedia working groups which handle shared (between several IMCs) services such as Listwork (lists.indymedia.org) do rarely receive a notice when an IMC - a local, open collective of self-organised activists organising open publishing media, especially through a website on the world-wide-web - either shuts down as a functioning group of people, and/or stops or suspends the normal post-publication moderation on its website. As a result, these IMCs' internet-based services often remain available, but are no longer maintained.

Depending on the type of service, this can cause the internet-based services to be abused, such as by email or blog spammers or for exploit hosting. Even if this does not happen, it is a waste of our limited resources. As such, this document "How to properly shut down an IMC website" is being compiled. Its audience is meant to be both
  • people who are shutting down their own IMC website for the variety of reasons there can be and
  • people who have become aware that an IMC website is no longer maintained (whether or not they have volunteered as an Indymedia activist before)
The document should be usable both by people knowing the global Indymedia structures and those who do not. It should be usable by any person whether or not they have a broad or just a small tech background.

It is intended that this document will be used by new-imc, imc-tech and all the working groups referenced within the document to notify the other relevant parties if they become aware of an IMC website which is no longer maintained.

HOW TO PROPERLY SHUT DOWN AN IMC

If you become aware of an IMC website which is no longer maintained,
or if you are an IMC that wishes to shut down your collective and/or
your website with a clear process, then please follow the steps listed
below.

*If you are external to the local IMC collective:*
 
1. *local check* Look on the website of that IMC and on
http://lists.indymedia.org for contact addresses (email) of this
IMC. Send an email to all of them, saying that you think this IMC
appears to be dead and state why you think so. Ask them to get back to
you within 14 days. If you receive no reply within 14 days or the
replies you receive support your impression of this IMC being
inactive, proceed to step 2.  Otherwise, stop here.

2. *regional check* Look on the list of IMCs on any IMC website and
http://lists.indymedia.org for contact addresses (email) of an IMC
that is geographically close to the IMC that appears to be
unmaintained.  Geographically "close" depends on how many IMCs exist
in that region.  First try the closest, then try more distant
ones. Or you can try the regional mailing list for that IMC:
http://lists.indymedia.org/mailman/listinfo/imc-canada 
http://lists.indymedia.org/mailman/listinfo/imc-us
http://lists.indymedia.org/mailman/listinfo/imc-latina 
http://lists.indymedia.org/mailman/listinfo/imc-europe
http://lists.indymedia.org/mailman/listinfo/imc-africa
http://lists.indymedia.org.au/mailman/listinfo/imc-oceania

If you receive no reply from any nearby IMC or the regional IMC within
14 days, then proceed to step 3.  If the nearby IMC or the regional
IMC mailing list agree that the suspected-dead IMC is no longer
functional, then ask that nearby IMC or an IMC in that regional IMC
mailing list to make a proposal to imc-process
(http://lists.indymedia.org/mailman/listinfo/imc-process) for shutting
down that IMC as a collective in the IMC network. Otherwise, stop
here.

3. *global check* Find *any* local IMC collective and describe the
steps you have taken (e.g. give URLs on mail archives) of contacting
the local IMC and regional IMCs. Explain why it was impossible to get
any response from the IMC itself and from the regional IMCs, and that
therefore, a distant IMC needs to make the political decision to
propose shutting down the apparently-dead IMC. Request that a proposal
(see 3.b) be made to imc-process to remove the apparently-dead IMC
from the cities list. If you are not active in any local IMC, then
stop here. If you are active in a local IMC, then either your local
IMC accepts to make a proposal to imc-process, or it does not.  If it
accepts, then after a 14-day deadline after the proposal has been made
on imc-process, go to step 5.

3.b The proposal to imc-process *should* contain details of what
archival plans there are, the website(s) and mailing list(s), of that
IMC.

*If you are from the local IMC collective:*

4.  Please post a message (typically, a central column feature) on
your website, or at least, post at least one message on your present
mailing list describing your collective's decision to shut down your
IMC, whether as a collective, or just to close down internet
services. Please send an email from your closing-down collective to
imc-process with a 14-day deadline (see 4.b). The network cannot
prevent a local IMC from shutting down, but a delay is needed so that
local collectives have time to cross-check information and understand
what has happened (or not happened). After the 14-day delay, please
take into account comments from other local collectives, made on
imc-process, and proceed to step 5.  If you (the local collective)
changed your decision and decided not to shut down, then stop here.

4.b The shutdown announcement to imc-process *should* contain details
of what archival plans there are, the website(s) and mailing list(s),
of that IMC, etc. If you wish to remain an IMC collective with only
a suspended website, then you should publish or ask for help in 
publishing a "we are closing + howto reorganise/restart" announcement
on your website before it is frozen and archived.

5. After the imc-process proposal has passed its 14-day deadline
(with no withdrawal from the shutdown by the local IMC in question),
send email to the following email addresses, giving a link (URL) to 
the imc-process proposal:

  new-imc lists.indymedia.org
  imc-tech lists.indymedia.org
  imc-commwork lists.indymedia.org
  listwork sos.indymedia.org
  dns sos.indymedia.org

6. If you have the ability to determine their web hosting provider(s),
please do so, and contact them, too. The general way to do so is to
determine their IP addresses from their IMCNAME.indymedia.org domain
name, then run a whois and DNS reverse lookup on these IP addresses,
and determine the tech collective or ISP from this. It can be useful
to submit the reverse lookup to a search engine, often there is more
information in mailing list archives or on docs.indymedia.org about
the hosting. Once you know this, determine their email address and
send an explanatory email to them, similar to what you sent in step
two.

Site lists

Hand tested, non existing when tested, nobody seems to defend these sites

Other sites status

There were some sites where the status has been clarified on the via people on email or still require clarification...

http://www.detriot.org/ - hacked?
http://mumbai.indymedia.org/ - hacked? -> [no isn't "baiacu" is a server in brasil we are hosting it we will keep it frozen for navegation on a new server - baiacu is having problems now]
http://india.indymedia.org/     [is not dead is froozen as a request from the collective - the server was down when you clicked on it]
http://chilesur.indymedia.org     [is not dead is froozen as a request from the collective - the server was down when you clicked on it]
tnimc.org/ - forbidden due to permissions problem, is there a site? (There is a collective, folks need tech help to relaunch -- Main.FatCity - 15 Jul 2007)
http://www.madhattersimc.org/ HACKED
http://utah.indymedia.org/ CLOSED SINCE OCT 2006
http://hawaii.indymedia.org/ NO PUBLISHING SINCE 2006
http://chapelhill.indymedia.org/ ONLY SPAM, NO EDITORIAL SINCE 2006 
biotech.indymedia.org - was seen up fairly recently but down now
boston.indymedia.org - down at time of test, now up again and fine
www.stlimc.org - no editorial features, collective reorganizing
hamilton.indymedia.org - closed to publishing but perhaps to return
melbourne.indymedia.org - publishing suspended, but perhaps to return
ontario.indymedia.org - 8 months of downtime. up now, just very slow or
timing out. only spam in newswire this year.
http://cvilleindymedia.org (this is just a template, no content -- Main.FatCity - 15 Jul 2007)
http://binghamtonpmc.org/ (They have a page up but have canned the site -- Main.FatCity - 15 Jul 2007)

Currently down or having problems, but with a more or less working collective:

  • kenya.indymedia.org - the imc is mainly doing offline projects, kwadronaut proposed to bring it back online without publishing.
  • video.indymedia.org - now accepting uploads after problem being noticed
  • radio.indymedia.org - hardware problems, being worked on

Computer generated 14th June 07

I've removed the list to avoid confusion but you can try out link validation of http://www.indymedia.org/cities.inc yourself using http://www.2bone.com/links/cgi-bin/2check.cgi

The computer generated results are not reliable as it doesn't give enough time for all sites to respond. Both Brazil and Mexico for example are routinely displayed as non existent yet they both work fine. Ontario usually fails also, timing out often even when hand checked. Oklahoma and Oceania are also good but show as bad below.

Frozen sites

Like mentioned above, there are some sites not being updated, but a seperate list of those could be practical:

History relating to cities.inc

I trawled up a few things of possible interest in the lists archives.

From http://archives.lists.indymedia.org/imc-tech/2001-April/003631.html IMC tech April 17th 2001 meeting summary

* Cities.inc - we discussed how this was used, why some cities were out of
date, how to update them. The master copy is on stallman in /www/cities.inc.
Cities need to symlink from local/include/cities.inc to the master copy,
once that is done, updates to that file will be part of your site. If you
dont symlink, then you have to update manually. If there is a really old
version of the code, or have heavily modified code, in which case
sidebar.inc doesn't use local/include/cities.inc then you need to contact us
to get that changed. We need to make a way for offsite servers to update
(possibly by a remote wget which checks once a week or something). 

We decided that changes to cities.inc should be done by
imc-process/imc-presentation. If someone wants a change they need to submit
that change to imc-presentation, who will decide on the change and then
change the file. When a new city is implemented, once process gives it its
nod, then they will add that list to the cities.inc. 

From http://archives.lists.indymedia.org/imc-process/2003-March/004673.html subject: remove central florida

It's been several months now since the hard drive crashed for the server
that hosts central florida.  We could recover it from backup, but
looking at the site we realise that it simply wasnt being maintaned -
apparently it was being run on automatic.  I sent several emails to
people and to lists, nobody replied, we finally put up a splash and
hoped that someone would contact us.

Nobody has.

I think it's safe to say that central florida IMC is dead, that it
should be removed from cities.inc.  If people (ie, James, Sharpie, etc)
want to get it going again that's cool, they should jump back in, but
from what I've seen that's not happening.  I feel that, at this point,
if a new group wanted to start the IMC back up again they should pass
through the new-imc process.

I don't know who does this, but can we get it down? It's embarrassing to
have a dead IMC sitting on the cities list for months on end.

...

This is the closest thing we have to "Dead Air", the most embarasing of broken links.  

It really should be pulled, and since the discontinuity of "ownership" seems to being a given and documented fact per Arc, its not too much to say any replacement must formally go through all the hoops , again.

-- BradleY - 22 Jun 2007 Under "frozen sites" I added: IMCs in the US that are Currently Inactive, etc...

-- BenUK - 14 Jun 2007

-- KwadroNaut - 16 Jun 2007 clarification about radio and kenya. Filled in the heading of frozen sitelist

-- ToyaMileno - 16 Jun 2007 added comments about india, chilesur, mumbai imc websites

-- FatCity - 15 Jul 2007 Addes stuff to IMC's in the US that are inactive
Topic revision: r20 - 26 Sep 2009, BouD
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback