<JpL> everyone present?16:15
<billf> yep16:16
* JpL pokes earthman
<JpL> pseudo can't be too far away
<PseudoPunk> I'm present
<JpL> :)

question #1 - do we need a facilitator?16:17
<billf> i don't think so i think you should just run with the agenda
<JpL> okay, that sounds good16:19

from the docs wiki:

# proposal - Defining an IRCOperator http://lists.indymedia.org/mailman/publi c/ircd/2003February/000077.html

# drone problems http://lists.indymedia.org/mailman/publi c/ircd/2003-May/000153.html


* drone network detection and placement in restrictive classes

* proxy/drone/socks/wingate/etc detection bot


# moving server connections to a non-standard port

# reducing split fallout: dedicated hubs? lazylinks?

# removing stale/flapping servers

# server requirements document (covering both hub and client)

# adding new irc servers

# creation of common ircd.conf include file

# common ircd process monitoring script

# using a dns rotation for all client servers

does anyone have any additions?16:20
<billf> nope
<JpL> anything I would add is simply a sub-bullet of one of those, so it will come up as necessary16:21
>earthman< are you around?
<JpL> just pinging earthman, we can give a few minutes
<billf> ok16:22
<JpL> plus I need some liquids
<sabcat> hell-o

whats this meeting about?
<billf> sabcat: http://docs.indymedia.org/view/Sysadmin/ IrcDMeeting01Jun2003Agenda
<sabcat> nice16:23

i've some serevrs

servers

and i'm also experienced in modding hybrid

so if i can help something then tell me

- proxy/drone/socks/wingate/etc detection bot16:24

i've already patches and stuff i wrote for this
<billf> well i'd suggest listening here and familiarizing yourself with whats going on
<sabcat> not a detection bot, built-in into ircd
<JpL> hang out and see if you can get familiar with the current state
<sabcat> ok :)16:25
<JpL> here, sabcat - I'll /msg you the agenda as it sits

that link for the first itemis bunk btw16:26

should be http://lists.indymedia.org/mailman/publi c/ircd/2003-February/000077.html
<sabcat> k

so i could add some I-line features for drone detection if u want
<billf> i fixed in the wiki a minute ago after i noticed it :)
<JpL> hehe, good catch :)

anyway, we can get started, earthman will return from the forest or not16:27
<billf> ok
<JpL> I would also propose that we limit the time to 2 hours

but I'm optimistic we won't need that much time anyway
<billf> i second, that should be more then adequate16:28

ObDisclaimer: this is the first imc meeting i've attended so slap me around if i'm unfamiliar with procedures
<JpL> also, if anyone has not read it, http://docs.indymedia.org/view/Sysadmin/ IrcDMeeting06Apr2003Logs may be useful to skim

the usual procedure is to have a facilitator and queue (stack) people up based on /me raises hand16:29
<billf> that log has as much entertainment value as it does technical substance :)
<JpL> "as much"? That's an understatement ;)
<billf> :)
<JpL> but since we are only 5 I would say we just discuss items freely - if it gets messy we will move to stacking16:30
<billf> ok
<JpL> if you have a proposal make it clear it's a proposal you're putting on the table and not just a comment/point of information

first item - Defining an IRCOperator16:31

the first question is whether we want to extend what was hashed out previously (and posted by micah in February)16:32

this is basically what has developed naturally
<billf> i think micah's document covers everything except gaining consensus on ircd@ before adding someone
<sabcat> basically when there r new servers new ops are required16:33
<billf> or if not consensus, just no objections
<JpL> aye, which in terms of mailing lists has tended to == consensus16:34
<billf> right

do we want to formally have a time period (48h?)?
<JpL> here, I will add this to the wiki so we can edit16:35
<billf> ok
<JpL> unless it is already there?

doesn't seem to be - brb
<PseudoPunk> 48 hours seem ok.
<billf> it's touched on in http://docs.indymedia.org/view/Sysadmin/ IrcOper

but more as a informative doc and not a hard requirements doc16:36
<JpL> actually there's a verbatim copy there
<billf> i edited that the other day to remove micah's des hash from the example :)
<JpL> hehe

oh yeah, I remember when that page was being put together16:37

question then - should we create a new page with a formal "requirements" listing or just work with that page?
<billf> i think we should just add the waiting period to the 'how to become' section16:38
<PseudoPunk> don't create too much pages.
<JpL> aye
<billf> the other question is if we want to formalize the difference between 'oper' and 'admin'16:39

or 'another question' rather
<JpL> if there are no objections to adding that 48h time period for blocks

then we can go with that as a proposal to be posted to the list

billf: where oper='ircd operator (o-line, operserv)' and admin='server sysadmin'?16:40
<billf> well, admin being someone with conf access and /die /rehash type access16:41

as of right now everyone who is an oper is also someone who has the admin bit set, but that may not make the most sense because a lot of those commands don't make sense without shell access16:42
<JpL> this also touches on what sabcat was saying previously

I don't think that a new ircd server necessarily implies a new oper
<sabcat> /die, /restart, /rehash are also dangerous without shell access
<JpL> in fact it never really has in the past unless someone who was already an oper set up a new server16:43
<billf> right, we should also discuss if being an oper and/or admin is a global thing or a per-server thing

in the context of indymedia, i would say that opers aren't and admins probably should be (but aren't)16:44

in the context of networks with autonomous servers (efnet, undernet, etc) everything is per-server
<JpL> (oh, another thing that might make sense even without real hard facilitation - when you finish a point/explanation/whatever close with '<END>')16:45
<billf> ok <END> :)16:46
<JpL> when you say aren't, what are you referring to? <end>
<sabcat> this raises the question if you want to control your network more than efnet, undernet & co.

for example to defeat cops & spies
<billf> well, as of right now i have access to che and can do things that effect the network as a whole, but can't make those same changes on the others. if we are going to keep a network-wide config, when an admin makes one of those changes (e.g. disabling half-ops or something) it should be network wide16:48

if we want the servers to operate independently, thats cool too. but if we want the 'admin' role formailzed and consistant policy that will require access and such <end>
<sabcat> thanx... btw. tell me if i disturb, i'm just talking into your discussion and you don't even know me...16:49
<JpL> sabcat: you're here, the meeting is open to everyone16:50
<sabcat> i'm from FAU switzerland and i've already some experience with IRC networks and the hybrid software

ok
<JpL> oh yes, we should have done brief introductions, but we skipped that due to so few people --
<sabcat> :)
<JpL> <-- Western Massachusetts, USA IMC, global tech group16:51
<billf> <-- san francisco, sysadmin group
<JpL> Having the ability to propagate changes consistently is a must-have16:52

I see at least two ways to accomplish this -
<PseudoPunk> <-- imc wvl/belgium,
<JpL> 1. provide accounts and access to the ircd installation (via sudo or whatever works for the sysadmin of the box) on an individual basis16:53

This should also include the ability to start/stop the daemon
<sabcat> how about ircd configuration file & ircd binary synchronizing? :D16:54
<JpL> 2. provide a shared 'irc-maintenance' account that

has access

With either of these it would be nice to have an automated synchronization script set up

as sabcat just suggested16:55
<billf> i dislike shared accounts in general
<JpL> as do I, I don't see that there is much to be gained, an in fact a sysadmin only loses the bit of accountability present with individual accounts
<billf> so if everyone who runs ircd servers is comfortable giving imc sysadmins (or if we define them, the subset that are imc ircadmins) access, that would work well16:56
<JpL> so the question is then - how do we determine who should have admin access and who should have oper access?

in the past it has been straightforward, anyone who wanted admin access had an account set up on the server

ahimsa2 has been an exception16:57

which has actually been a problem, since things have gotten out of date on said server

due to lack of provided accounts
<sabcat> we handle it like this on our (small) network: admin is global, and his ssh public key should be on all ircd accounts16:58
<PseudoPunk> there are not much people with a personal ahimsa account.
<sabcat> so admin is an allmighty fucker

:p
<JpL> sabcat: do you have a shared admin account with multiple ssh public keys for different people?16:59
<billf> i would suggest that the difference between an ircd admin and oper is one of policy v. day-to-day ops
<JpL> really what we seem to be discussing is another 'requirements' doc, one that outlines what a server has to provide when they join the network
<sabcat> JpL: yes, but there are only 2 admins, but every ircop knows their cell phone so they can ./ircd in an emergency
<billf> yes, i would agree that 'access to the ircd admins' would be on that document, but we haven't defined who or what they are yet17:00

access for the ircd admins, rather
<PseudoPunk> but if admins have accounts on all servers we need to be damn sure who we make admin17:01
<sabcat> yeah but even your admins you have right now with the +A flag

they can send remote SQUIT and shut down the whole net too17:02
<JpL> I'm feeling at a loss as to how to define it - all that comes to mind is that admin access can be requested after being an Oper for some period of time
<billf> yes, which is why i don't think we should make every oper an admin. i'd even go as far as saying its a manageability problem and not a heirarchy type thing
<JpL> sabcat: yes, right now that is the case

hence the discussion about the distinction
<sabcat> JpL: i'd say admins should be these people who have also physical access at least to the hub... as our 2 do
<JpL> this is partially for my own sense of continuity - summary-wise17:03
<billf> well physical access isn't as big of an issue as much as shell access
<JpL> 1. do we draw a distinction between oper and admin? (seems to be a yes)

2. do we require new irc servers to give admin access as necessary? (seems to be yes)
<sabcat> on our net an oper is defined as someone who can act on protocol level only17:04
<JpL> 3. is oper access global? (seems to be yes)

4. is admin access global? (not clear)

5. how do we determine who has admin access? (")

right now admin access is not global, it is based on who the sysadmin is willing to give an sudo account to17:05

and as PseudoPunk said, ahimsa is short on accounts right now, making it difficult to maintain

I believe everyone who is an oper has admin access to kropotkin and che (and judi before it was out of the network)17:06
<PseudoPunk> i don't
<JpL> okay, was just going to check for sure :)

do you have admin access to any of them?17:07
<billf> i would agree with "yes" for 1, 2, 3. #4 should be strived for in all existing servers (even if it takes a while to roll out) and a requirement for new servers

#5 should be those people who run an individual server + anyone involved in the policy/conf/backend
<JpL> which basically means someone says "I am or want to be involved on that level" and then we work by consensus?17:08
<billf> yes, i think the same 48h vetting period in the working group would be appropriate - although i would say that the current admin group would be the people who could block/propose17:09

for obvious reasons, the initial group would have to be blessed by the working group as whole17:10
<JpL> that makes sense - were we making it so that only current opers can block/propose new opers?
<billf> i don't want to introduce too many levels of hierarchy or anything along those lines but there is something to be said about "too many cooks in the kitchen" and so on17:11
<JpL> also it will give a larger comfort level to those who have future servers to offer
<billf> i would guess that a block that came from someone uninvolved wouldn't carry much weight if existing admins/opers voiced their support in lieu of the block17:12

but i don't know how to formalize that feeling :)

or even if it should be formailzed
<JpL> the only other specific point that comes to mind is whether one should have the ability to be an admin on some subset of servers17:13
<billf> (unofficial point of order: we're approaching 1h and are on the first agenda item)
<JpL> yeah, I was just noting that as well :P
<billf> i guess if an admin is uninterested in working on the global network, they don't have to
<JpL> I can foresee a case where someone donates as server and becomes an oper, and hence also have admin access on their server, for instance17:14

we also had a model at one point where a particular individual was 'responsible' for each server (well, actually there were two active servers, I had one and Deanna had the other)17:15

but the hope was to distribute that more widely
<billf> right, that goes back to the efnet/undernet/etc model - which works as well as long as servers don't have conflicting policies, but i would think since we control most (all?) of the machines in question and are providing this as a unified service that we want things a little more network-wide17:16

back to the donation scenario, if that person isn't proposed as an admin to the list, i guess that informally makes them a server-only admin
<sabcat> i want to donate!

:)
<JpL> so trying to move along, does anyone have any major issues with what has been presented? I can form it into a concrete proposal/addition and post it17:17
<PseudoPunk> sabcat: no problem :-)
<billf> JpL: i would agree with that17:18
<JpL> agenda-wise, we can either go to drone problems or discuss addition of new servers (as it is related)17:19
<PseudoPunk> I agree
<billf> lets hit the new server document, the drone problem is standalone
<JpL> currently the servers involved in the IRCd network are: judi, kropotkin, che, and ahimsa217:20

judi has been dropped due to flaky adsl connection

kropotkin's connection is a not a whole lot more reliable

at one point we had 6 servers total, but there were delays and some of them were lost for whatever reason (usually unresponsive admins)17:21
<sabcat> i have a reliable connection

located in bern/switzerland

2gbit backbone of sunrise
<billf> sabcat: we're just discussing the requirements not specifics
<sabcat> oh ok.. i c17:22
<billf> or are we?
<JpL> my questions would be - what are the requirements for new servers? how do people make offers to donate?
<sabcat> maybe you need a more reliable hub :)
<JpL> again, for historical reference, servers have come up as potential during informal discussion17:23

they have all been offered thus far by someone who either runs that server for another IMC purpose or at least does work on another server17:24
<billf> my requirements wishlist (in no order): shell access for global admins, agreement on ircd wg, "professional" connectivity (avoid residential dsl, etc. prefer colo'd machines)

machines that are offered for use should be running an up-to-date modern unix
<JpL> can you define 'agreement on ircd wg'?17:25
<billf> well i think that new servers should be 'mentored' in by an existing admin rather then making it an 'application' process17:26

not that we should shun donations, but given the sensitivity issue (we're basically giving them a irc sniffer), there should be some sort of comfort with who we're adding
<JpL> that sounds like a good plan17:27
<billf> if we make a requirements doc, we'll get a lot of people who want to run a 1 client server

the efnet new-server-guidelines.txt is a good reference as well, though for a network of this size some things don't apply17:28

it covers things like ntpd, how a machine resolves hosts, questions how the machine is protected (wide open, firewall, etc)17:29
<JpL> I'm comfortable with the mentoring model and the requirements we've listed
<billf> some of the things there are a bit strict so it can't be used at all verbatim, but it wuold be a good reference to someone making a requirements list17:30

compiling the ircd to the specifications in the wiki is also something that should be listed
<sabcat> ircd should be compiled with dietlibc :P17:31
<JpL> yes, those specifications and procedures were put together to help make the addition of new servers a bit easier

http://docs.indymedia.org/view/Sysadmin/ IrcdSetup17:32
<billf> yep
<JpL> and conistent

consistent

if anyone has specific suggestions for changes or streamlining (there are a few in the agenda that we'll hit), by all means bring them up on ircd@lists17:33

related agenda item #1 - rotating dns
<sabcat> how about certificate verification (against MIM) on SSL connections?
<billf> we already have a source of truth for the ssl certs on che17:34
<sabcat> yes but hybrid currently does not support verification of signed certificates against a CA
<billf> right, which is why we distribute the public half of the key
<sabcat> oh ok17:35

i have a hybrid patch for global CA support
<billf> i would say that any machine that is providing client service (which means they MUST be running the stunnel stuff) should be in the rotation
<JpL> billf: was the 'server requirements document' agenda item covered suitably for you?17:36
<billf> yes, are you going to formalize that into a proposal or would you like me to?
<JpL> I have some points listed, but if you can formalize it that would be great
<billf> ok i'll do that
<JpL> rotating dns is related to the hub/leaf/lazylink item17:37
<billf> yes, i was just going say something about that

i don't think that the bursts and traffic on this network is enough to go to lazylinks17:38
<JpL> the 'new server' part of it is simply that new servers should be linked and some period of testing should pass before being added to rotating dns

right, lazylinks have not seemed to make sense
<billf> but i do think we should have two dedicated hub ircds
<JpL> the other question is topology - right
<billf> i think a 30 day 'trial connection' period would be adequate
<JpL> that basically makes a spanning tree though

and do we want that rather than a star topology?17:39
<billf> well if we have servers whose network connectivity is sufficiently better then the rest of the nodes, i think introducing dedicated hubs makes sense
<JpL> but the question is one or two17:40

what benefits would we get from having more than one center hub - since only single routes are allowed in hybrid717:41
<billf> failover, mostly

it does introduce the problem of having two distinct networks when one of those hubs splits

but i guess that goes back to choosing well connected hubs
<JpL> right, that is my concern, and you don't have that problem with one hub
<sabcat> billf, its good to have a backup hub17:42
<billf> wel, then we have 5 distinct networks

and if one hub goes down, all the client servers should jump ship to the other one
<JpL> well, we'd have more than two distinct networks with two hubs
<sabcat> i also wrote a hybrid patch here stick out tongue it will connect to failover hub when it looses the main hub, and will recognize when main hub comes back and then relink

(i'm talking of one hub which isn't connected normally)
<billf> relinking causes additional churn
<JpL> (if one went down with 4 nodes, the 4 nodes are then separated, in addition to those on the other hub that are still linked)17:43

hmm
<billf> those 4 nodes should autoconnect to the other hub
<JpL> is that possible with stock hybrid7?

or just with sabcat's patch added
<billf> stock hyb17:44

if multiple servers are configured for autoconnect, hyb will use them
<sabcat> yes hyb will use them all at once
<billf> so in the case of one hub dying with all the current client servers, they'd all move to hub#2 isntead of making 4 islands17:45

in the case of a mesh topology where all servers can connect to each other, it's a crap shoot as to how the chips fall after the primary server dying
<sabcat> billf, you can't keep stock hyb to not connect to hub#2 when hub#1 is there, he'll try to connect anyways..17:46
<billf> only if there is no connection
<JpL> okay, that was my question, whether you just rely on the multiple routes causing a failure or whether you set a priority
<billf> i don't think we need a distinction between primary hub and backup
<JpL> sabcat: it won't connect successfully to the second one since there is already a route to the first17:47

unless I've missed something about hybrid
<billf> if you have two well connected hub servers it really shouldn't matter where a client server connects to, though obviously if clientservA and hubA are topologically close to each other you'd prefer that
<JpL> won't the server keep trying to autoconnect every X minutes to the second hub?17:48
<sabcat> JpL: yes sorry i was wrong about that, when compiled as leaf it will not even try to connect when there is already 1 link
<billf> right, and in hyb7 its runtime not even compile, so in a true emergency/meltdown we can shift the topology around17:49
<JpL> compiled as a leaf or configured as a leaf?

aye
<sabcat> JpL: i was in doubt about that, but now i'm sure, hybrid will disable auto-connect when linked and compiled as leaf

compiled

config.h

#undef HUB
<billf> sabcat: nope, serverinfo { hub = no; };17:50
<JpL> sabcat: which version of hybrid?

almost everything is runtime now
<sabcat> ah ok, i was thinking about 6.x17:51

lol

you're right it changed in 7.x
<JpL> anyway, to get back to the question at hand17:52

currently ahimsa2 and che have reliable links, but we can't really make them both dedicated hubs because that leaves only the unreliable kropotkin as a client node, which defeats part of our purpose
<billf> so back to the agenda, i think that after we have a formalized admin structure and config distribution and such, we should investigate dedicated hub servers and/or run a second ircd on one of the existing machines
<JpL> heh
<billf> jinx :)
<JpL> :)17:53
<billf> well, i have several very well connected machines (currently serving sf.indymedia.org) that could be used for this
<JpL> we began setup on one of those machines once upon a time I think17:54
<sabcat> me too

:p
<billf> yes micah and i discussed it and i had it ready to rock'n'roll but that was around the time of political hell

so we tabled it
<JpL> yep17:55
<billf> sabcat: i'd entertain a european hub once we get this initial stuff off the ground
<JpL> well, for now, as in immediately - we have 3 servers17:56
<billf> i think it would best to get the other documents formalized first
<JpL> I would propose we keep che as the hub, but it still accepts clientsd
<billf> yes this is something we can gradually do
<JpL> Are there any reasons to work with anything other than a single hub immediately?17:57
<billf> immediatly? i don't see impending doom or anything.17:58
<JpL> if not, are there any immediate proposed changes to the topology?

just thinking in terms of things that need doing asap17:59

or rather, things that can be done asap without other criteria that need to be met
<billf> no, i think the formalized documents and templating the .conf stuff is a pre-requisite to moving to n+1 hubs
<JpL> yes, which brings us to the next item - ircd.conf18:00
<billf> the stuff in che:/root/ircd_template is a bit stale relative to che's current config

my plan was to break them up into imc.opers.conf imc.servers.conf etc
<JpL> Yep. I did go through and trim down the conf file to a manageable size, but you just noted the problem with that
<billf> and then provide an ircd.conf that only contains the local stuff18:01

and distribute the .include'able stuff

which makes merging changes much easier

also using rcs etc etc
<JpL> I'd say that breaking it into .include parts is a great idea - what do we want to set up for distribution and rcs-wise?18:02
<billf> i planned on making it available using rsync collections
<JpL> heh, one step ahead again
<billf> with hosts allow control and such
<sabcat> how about cvs? would make less trouble than rcs+rsync..
<billf> i don't think they belong in any larger cvs archive because of the sensitive nature of some of the values
<sabcat> why larger cvs archive? one could dedicate a cvs to it18:03
<JpL> sabcat just echoed my thought

unless there is another reason to not use cvs here
<billf> pserver or over ssh
<sabcat> over ssh with public keys18:04
<billf> that's overly complex
<sabcat> or ssh with host-based auth?
<JpL> i just realized it's been a really long time since I used RCS that wasn't integrated into something else :)18:05
<billf> cvs introduces things like merge conflicts18:06

and i don't think we need remote commits

in fact, i'd say we shouldn't
<sabcat> hmm yeah u're right, probably some directory containing RCSed cfg files and an rsyncd serving it would be better18:07
<billf> it keeps the authority on the server that has the files instead of distributing it18:08
<JpL> so if we went with rcs and rsync, updates would have to be run on the remote machines - manually or via a frequent cron job?
<sabcat> cronjob is dangerous, the cronjob would also rehash ircd and this can lead to a parse error.. worst case: ircd dies18:09
<JpL> manually meaning either from the remote machine or via a script that started rsync on all servers immediately via ssh
<billf> i think the script should be cron'd and that'd still allow for someone to manually run it18:11

there may be value in making all the cron's not run at the same time though

presumably, che's ircd.conf could .include them from wherever they live and provide a staging area18:12

it'd have to be quite a fucked up config to make the ircd die though
<sabcat> indeed

hehe
<billf> <end> :)
<sabcat> i need to leave you soon, so what i'm most interested to help is the drone/clone/proxy/etc. stuff18:13
<JpL> okay, the ircd.conf can be broken up and we'll try using rsync and rcs to keep consistency across servers
<billf> automating it would also workaround the problem with machines that for one reason or another can't give out accounts to all admin <end for real>
<JpL> we're coming up on 1:45 or so time, let's see if we can get through some other items18:14

billf: yeah, my thought as well, though we will still attempt to get access for all admin as a requirement
<billf> ok the ircd process monitoring is simple, i've got one for services and the tcm working already, i'll make an ircd one available soon <end>18:15
<JpL> I saw that servchk was running now
<billf> yeah as user irc instead of uid 0 now too
<JpL> yep
<billf> next topic i think requires discussion kroptokin18:16

kropotkin too
<JpL> we could shift to drones instead for sabcat's sake18:17
<billf> ok lets do that
<JpL> Point of Information: we have a drone problem :)
<sabcat> i already saw that ;)
<JpL> billf has done a bunch to improve the situation18:18
<billf> i'm interested from hearing from the group if we are willing to use active monitoring (probes, etc) for detection
<sabcat> maybe some fuzzy logic could be useful there, to detect nonsense patterns in nicks
<JpL> can you briefly outline what you have in place?
<billf> ok currently, there is a script that parses the ircd logs and tries to find connections that are exceeding the conn/ip limit18:19

it can produce either a dline list or a i:line (er, auth block) for these rogue machines and it can also aggregate them into networks
<sabcat> i would suggest passive monitoring18:20
<billf> that's also occuring with a hybrid-tcm bot
<sabcat> to incorporate a new feature into i-line
<billf> which currently takes no action except to alert, but soon will start adding new dlines

once i'm comfortable enough to put it autopilot

and i'll notify ircd@lists once this occurs along with a brief how-to on controlling it18:21
<JpL> example spam:

[17:18:48] <chemon> Possible clones from AC844868.ipt.aol.com detected: 3 connects in 4 seconds

[17:18:48] <chemon> * clone violation from bes_f2957 (~6164@AC844868.ipt.aol.com): No actions taken

[17:18:48] <chemon> bes_f8151 is ~5598@AC844868.ipt.aol.com (14:18:39)

[17:18:48] <chemon> bes_f2957 is ~6164@AC844868.ipt.aol.com (14:18:38)

[17:18:48] <chemon> bes_f9270 is ~255@AC844868.ipt.aol.com (14:18:35)
<billf> eventually, i'll be 'retiring' the dlines into restrictive classes

and the retired dlines will still score into a 'bad' network

the problem with leaving the dlines in is 95% of these are dialup/dynamic broadband/etc and a month from now the infected machine has a different ip18:22
<JpL> ah, how are you tracking them?
<sabcat> a connection throttling algorithm && fuzzy pattern detection would work great
<billf> exceeding the conn/ip, continued attempts to connect after dlines, tcm alerts18:23
<sabcat> billf: it would enlarge config files like hell... even the config chain of the ircd in memory, ircd will become slow in finding i-lines..
<billf> by putting them in the restricted class they'll only have one connection/ip

sabcat: that's why i aggregate rogue networks into /24s

i have a fair amount of experience (large client count efnet servers) in being able to classify good clients before it hits the badguy blocks18:24
<sabcat> ok18:25
<billf> and that has been a successful approach: good clients (stunnel, working ident, whitelists), known dialup/broadband clients, restricted networks (built by scoring them), and then identless clients

matching in that order18:26

18:18 <billf> i'm interested from hearing from the group if we are willing to use active monitoring (probes, etc)

for detection
<PseudoPunk> I go to sleep. cu all18:27
<billf> however, to get the ones that only connect 2 drone clients (fall under the clone radar) or are in the restricted class so they can only connect one, it gets a bit harder

although if they're in the restricted network and they repeatedly try and connect 10 more, the tcm will pick that up too
<sabcat> a solution in the ircd code itself would prevent us from seeing annoying message when these drones connect
<billf> i only bring it up here because it is more invasive then log parsing and all these other passive techniques18:28

and want people to be comfortable with having our ircd machines making outbound connections to potential clients

i'm trying very hard to avoid modifying ircd code, i think hyb provides enough information in logs and to opers (and hence, the tcm bot) that it can be done without maintaining a divergent codebase

and if opers don't want to see annoying messages, there are plenty of umode flags (or just modify your client..)18:29
<sabcat> so i'm offering an ircd patch which allows a feature in I-lines that does connection throttling and detects random nicks of these drones before any outbound connection (identd, proxy) is made..18:30
<billf> if it is compilable as a module and passes a code review by ircd@lists, that may be a possibility

but there is a lot to be said for running stock code18:31
<sabcat> ok i understand u in this point..18:32
<billf> so is it a diff to the code or is it a module?
<sabcat> i could let other peoples to review/evaluation..

does hybrid 7 support modules?
<billf> on operating systems that support dlopen() yes
<sabcat> yeah should be possible to do this as module... depending on the callback system of hybrid718:33
<billf> the detection logic is more important in my opinion then connection throttling
<sabcat> it will go hand-in-hand18:34

if similar nicks connect in a short period from same subnet
<billf> well we don't have resource problems because of this, we just want to avoid the abuse

and without the connection throttling, it could be a tcm module and cause a lot less problems if something goes wrong18:35
<JpL> sorry, had a visitor arrive, was afk for a few
<billf> no offense to your coding abilities or anything, just more of a programming theory thing..
<JpL> just reading backlog18:36
<sabcat> i understand your doubts, billf... stock hybrid is a trusted/stable source and shouldn't be changed if theres another solution18:37
<JpL> do we want to expire the addresses placed into the restricted class?
<billf> JpL: thats my plan
<JpL> I mean expire them out of that class back to 'users'18:38
<sabcat> i would also do tcm module instead
<billf> i have no code to do that currently, but if they don't appear to be hammering at the server after they're placed in the restricted box that would work18:39
<sabcat> :)
<billf> if they're in the restricted class and they try and launch 20 drones, the tcm is going to see those failed attempts and dline that specific ip again, time will pass, back in to the restricted class, repeat..18:40
<JpL> right
<billf> the trick is not having things oscilate too heavily

which is just a matter of tuning
<JpL> active vs passive hasn't really been addressed18:41
<billf> nope :)

we're already doing ident, but that's a function of ircd. this would be connection with the intent of probing.18:42

given sufficient documentation (motd, wiki pages, etc), i personally am comfortable with it

but i could see how others wouldn't be
<JpL> which is done by a fairly large percentage of servers elsewhere anyway
<billf> yes.18:43

users who have working ident or use the stunnel wouldn't get probed either

it can be done on a per-class basis, so we could even go as far as only probing the restricted class
<JpL> my personal leaning is that if everyone here is comfortable with it (meaning the three active folks), we bring it up on the ircd list as a proposal18:44

for my part I think it would be fine as defined
<billf> ok, once i get the more extensive class/auth blocks done (so that 'real' users hit those first) and see how well that matches our client load, i'll propose probing the questionable clients on the list18:45
<JpL> it's not clear to me how many of these drones are using open proxies
<sabcat> assume they're not using proxies, they could be even some win32 viruses
<billf> i have done no research into fingerprinting the os of the drones or determining if they're IIS/SQL vulnerable machines or whatever

so like many security things this is just one component of stopping them18:46
<JpL> the nick/user patterns aren't specific enough to glean much either

I did notice that we're not in the list of irc servers in the original Fizzer worm though :)18:47

I was going to try to interact with one of them more closely the other day, but bill was too good at getting dlines into place :P
<billf> yes, a lot of our users would match any generic regex we came up with. one of my friends is fooling around with the nicknames/usernames of the drones that have been connecting to see if he can write some logic to detect them18:48
<JpL> I was thinking of more actively engaging with irc-unity, or at least seeing if someone wanted to do so18:49

though it's not clear whether that group is going to implode before it becomes a general purpose resource
<billf> i'm not familiar with them
<JpL> they addressed the Fizzer worm, and are focusing on these precise issues. there are representatives from many, many networks - url...18:50

http://www.irc-unity.org/
<billf> oh those guys (thank you google)

and thankyou jpl :)18:51

well they make my active socks probes look like a nonissue

we're approaching 2h30m18:52
<JpL> i missed something that caused the disclaimer on http://www.debugoutput.com/fizzer/

yep

so this is offtopic

there were really just two more items after drones, I think the drone discussion will be ongoing

but it seems that you should continue to work on what you've been doing, if you need any assistance just ask18:53
<billf> yes, and things like distributing the foo.conf files will help the finer grained auth/class stuff
<JpL> yep
<billf> although 92% of the clients are on che.. :)
<JpL> the question of active vs passive should be brought to ircd@lists

yeah, well, that's the rotating dns bit :)
<billf> haha yea :)
<JpL> (the funny thing is that people are still on other servers, but anyway)18:54

there were two other items:

# removing stale/flapping servers

(kropotkin...)

and

# moving server connections to a non-standard port

the link between ahimsa2 and che has been good
<billf> the second one is easier, i just think that we should connect the ircds on port 7000 isntead of 666718:55
<JpL> keeping client connections on 6667 I assume
<billf> yes

its just a "nice to have" item for debugging/firewalling/etc
<JpL> other than for the sake of ... okay18:56

I was trying to think of other reasons, but that's sufficient really

no reason not to that I can think of anyway

and what was the focus of "# removing stale/flapping servers"18:57

?
<billf> do we want to add it into the rotation given how often it splits18:58

and if not, what then is its worth as a client server
<JpL> not much imo - realistically all nodes other than the (eventual) dedicated hub(s) should be in the rotation18:59

unless they serve a specific purpose, temporary or otherwise
<billf> i guess it's a "someone had to say it" items :)19:00

i have no personal vendetta against it, but if we have other servers available i don't know what purpose a server with 8 clients is
<JpL> I spent far too much time tweaking judi and kropotkin to try to get them to stay linked for a reasonable period of time19:01

so I'm close to having a personal vendetta against their uplinks at least :)
<billf> as far as network topology goes, che and kropotkin are fairly 'close'

which makes their inability to hold a connection even more confusing or at least makes me more willing to blame whoever is controlling the infrastructure at the uplink19:02
<JpL> the router in front of judi was an issue last year, that was solved by lowering the ping time19:03

I am at a loss as to what the problem with kropotkin is other than that the adsl link must just drop frequently
<billf> right that can help but often it just means the connection is lagged or bursty instead of splitting, which is better but still not quality service
<JpL> in the case of judi it solved the problem since the router was dropping idle tcp connections (30 seconds or something extremely small like that)19:04
<billf> stateful firewall rules on the router?

and i read "lowering the ping time" as "raising the ping time"
<JpL> in kropotkin's case after a split I'll often see 50-75% pl, but I usually miss the splits immediately
<billf> so yeah that makes a lot of sense19:05
<JpL> ahh, okay

so I'm inclined to think that in the case of kropotkin it's just a flaky provider

but if anyone has any suggestions otherwise, by all means ...
<billf> i can't think of anything to get around that19:06
<JpL> (and despite pacifying judi's router, the link itself was shoddy anyway)

at any rate, this is covered in our server requirements now
<billf> so i guess that leads to "how unstable is too unstable"

right
<JpL> ahimsa and che were specifically brought up since they have nice links19:07

and, we're a bit past 2:30 - I'm going to be getting dinner with folks in a bit, don't know about other folks19:08

Does anyone have anything burning to add at the moment?
<billf> nope. are you going to post the log of this or should i
<JpL> I can do so before I head out.
<billf> ok that'd be great19:09
<JpL> right-o, I'd call this adjourned then. lots of signal this time around. :)
<billf> yeah, funny that.. :)19:10

Topic revision: r2 - 02 Jun 2003, JpL
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback