anti-floodnet algorithm

[Phil^]

After yet another floodnet hitting #planetarion im starting to get a little fed up of them, and the rate at which the channel bot, pea is able to detect and remove them
therefore i, out of boredom decided to see if an anti-floodnet algorithm could be devised which could detect them more rapidly

The main characteristics of a floodnet typically are :

1. Lots of clients joining a specific channel in a short period of time, usually in excess of 10 every 2-3 seconds if sufficient bots are available.
2. the bots usually have similarly themed names, often in the [a-z][0-9] pattern where there are multiples of either part
3. the bots usually come on to do something, such as say a specific string , notice the channel or just to part straight away again - they dont just join and sit there usually.
4. the bots usually dont have any patterns in the ip addresses used to do the attack. Mostly zombie'd machines.
5. sometimes, but not often the ident of the bots is the same, or follows a similar pattern to the nickname

So, possible means to tackle such bots

• when people join, note the time they did so, and compute the joins per .5s ratio.
• generate nickmasks from their nick and ident, so a person with nick "absdgs[453]" would have a mask of ccccccsnnns , where c = character between a-z , n = number between 0-9 and s - symbol being everything else.
• generate a wildcard mask in addition to the nick/ident mask to catch variable length nicks for bots, so the previous example would become c*sn*s , where the * represents n characters of the previous char type

the ratio of joins per .5s can obv catch when lots of people join at once, it may be necessary to include a joins per 1s, 2s in addition to it though
the masks can go into a hashmap or something and be used as primary keys. every time a nick with the corresponding nick/ident mask joins, the total is incremented , and a ratio for that mask created for the joins/time periods

if a certain mask goes above a certain point ( with the wildcard masks having a higher threshold trigger, especially c* ) then set mode +mr and start banning those who have joined within the last 10 seconds and match the nickmask
if there is more then 2-3 bots who joined within the last few seconds, and said the exact same message / did the exact same thing then set mode +mr and start banning those who match their nickmasks and recently joined.

now, this is just in the doodling down stage but any comment, suggestions etc would be appreciated

[Anonymous Hero]

The floodnets could use a highly unique set of nickmasks to prevent all of this and make it entirely useless.

The easiest thing to monitor is the rate of actions from non-authed users. If there are too many joins/notices/msgs from people without a P auth, lock the channel. Anything else seems as if you'd be spending lots of time creating algorithms which could easily be avoided with slightly more randomisation.

[Phil^]

yup, it can be avoided through randomisation of nicks, but the mass join trigger part of it is essentially all we have atm, and that wouldnt be evaded. its just a means of speeding up the detection of 99% of them out there
non-authed people would be tricky to check if they dont +x, and the channel isnt set +r since it would involve messaging P to check if they are logged in or not, and if a lot join quickly it would get P to ignore the detector from flooding it, or by /whois'ing them as they join for the relevent whois return raw message from the server, and too many of those and you get kicked from the server for flooding

[Anonymous Hero]

Most of the actions of a floodnet are usually done within the 1st minute?

Users who have joined the channel for 60 seconds or more could be added to the list of users which don't need analysing. Maybe rate limit the actions of all non-authed users who have joined in the past 60 seconds.

[Phil^]

after 60 seconds, the 'damage' done by the floodnet is already done unfortunately, but yeah otherwise that would probably prevent flooding off verifying nicks
its much more preferable to catch them in their early stages, ie before more then 10 bots join and flood. preferably around 5

[Ramihyn]

A few random comments:

I dont think trying to find nick/ident/user-masks and acting upon them will help you because its very trivial to counter. Same for the content of what the zombies say. So i wouldnt invest much or any time into that.

If you try to solve this problem on a irc-client level like another bot or an addition to the existing bots, you need to run your solution either as service or at least have it use DCC CHAT connections to communicate with other bots like P/Pea. Otherwise you will inevitably run into own flooding problems due to the nature of a floodnet/zombie-attack. Ofc. you could try to counter it by doing the same as the zombie-network - use multiple own irc-clients and have them communicate outside the IRC server with each other but that might still get messy.

A more serious solution could probably be done on a irc-server level with an additional flag for a channel. Because the irc-server could catch and avoid those attacks before they even show up for the other users/channel-ops. A simple example would be to suppress mass-joins into channels with a specific channel-flag set and suppressing channel-floods by applying the same flood protection to all users (beside ops and authed persons) as its applied to individual users now. Obviously the values would need to be tuned. I assume such additions to the irc-server code already exist so you would just need to talk with the netgamers irc-ops.

[Anonymous Hero]

One script I remember seeing analysed the nicknames of users who joined and kicked people with useless combinations of letters. An example of "ejqxwdgr", the amount of users with a *qx* in the nickname is probably very rare, also the fact that theres only a single vowel and it's at the start of the nick also shows its not a genuine user.

You should also be able to analyse the history of the channel for all users. If say, [email protected] has been in the channel before and wasn't kicked, then the chances are they're not much threat. However if too many users which have never joined the channel before join in a short space of time, then theres going to be a problem.

[Ramihyn]

Originally Posted by Anonymous Hero

One script I remember seeing analysed the nicknames of users who joined and kicked people with useless combinations of letters. An example of "ejqxwdgr", the amount of users with a *qx* in the nickname is probably very rare, also the fact that theres only a single vowel and it's at the start of the nick also shows its not a genuine user.

In which language? Are we discriminating polish, russian, danish, norwegian, klingons and many other types of chatters again? ("Ernst" for example is a quite normal german name)

Originally Posted by Anonymous Hero

You should also be able to analyse the history of the channel for all users. If say, [email protected] has been in the channel before and wasn't kicked, then the chances are they're not much threat. However if too many users which have never joined the channel before join in a short space of time, then theres going to be a problem.

What happens during events like creators hour, end of round or alliance competition etc.?

Both measures could be countered by a floodnet script anyway. They could probably help you for 2 weeks and then you are back to where you started.

[xtothez]

Looking at the original post, I think the solution lies in halfway between that and the method used to flag spam emails.

Certain characteristics of a client could contribute to a points total:

• The nick matching a botlike pattern ( i.e. CCCCC-NNN )
• A shared ident between several clients adds to all their points.
• Attempting to notice the channel or say several lines just after joining.
• Presence of a .users.netgamers.org vhost removes all points.
• Joining within X seconds of other possible candidates increments all exponentially (based on how many join).
• Plus, if this anti-flood bot was in multple channels it could flag known floodnet clients if they had been seen elsewhere in the network.

It would be relatively simple to combine this detection into a new bot that behaves in a simlar fashion to Stats/Pbot and the like, and takes orders from channel ops to list flagged clients and have select preset reactions to certain points limits (setting +m/+r, kicking the clients, etc).

[Jeekay]

I have been wondering about something like this, and pondering some sort of programming competition to kick off interest in this area.

If anyone wants to have a (serious) go at this, I'd be more than willing to help in any way I can. Ideally it'd speak P10, but that's not really a requirement. The way I figure it, if the algorithm is nicely encapsulated anyway it won't really matter what the underlying connection mechanism is.

Anyone willing to step up to the plate?

[Markb]

I've been working on fireclaw ( Pea is a fireclaw bot) for a couple of months now, one of the things that I was going to add in, is that if there have been more than <x> channel notices in <y> time then it +mr locks the channel.

One of the possible reasons that Pea takes a while to ban a floodnet is due to the anti flood function in the irc servers. Would connecting Pea to the floodless server that Stats / Pbot are on make the bans pretty instant?

[Alki]

what are the purpose's of floodnets, asin what does the 'attacker' get from it.

(just trying to educate myself)

[xtothez]

Originally Posted by Alki

what are the purpose's of floodnets, asin what does the 'attacker' get from it.

(just trying to educate myself)

They're generally used for one of three things:

• Disrupting channels or servers with tons of spam, preventing people from using it. Essentially a targeted denial of service.
• To attack users or channels so the person in charge of the floodnet can 'win' on the Internet. The teenage script kiddies who typically employ floodnets think this makes them 'leet'. The floodnet is a s'kiddies attempt at having the last word in a disagreement.
• To advertise URLs containing dodgy content such as porn, ads, warez, and often delilver malware via browser exploits. Several bots spead this way, making the floodnet larger.

[cisco]

surely a simple join-flood or clone-flood script that locks the channel +mr will do the job adequately?

(How did you get on btw Phil^ di u solve anything?)

[Ramihyn]

Originally Posted by cisco

surely a simple join-flood or clone-flood script that locks the channel +mr will do the job adequately?

I wouldnt call it "adequately" as it results in a disruption of the channel and thats exactly one (or _the_) main reason for "floodnets". It would be slightly less disruptive if the "legal" users would automatically get +v, but that again raises other flood problems.

[-TPM-]

How about a simple challenge - response? It could be a simple yes/no question. If the joining party doesn't answer yes/no or answers incorrectly within 30 seconds they get booted. You could also have a known user list, and not bother asking them.

[pablissimo]

It's not language-neutral, and as PA has shown even reasonably innocuous questions can have multiple answers.

Actually no scratch that, the PA bot-stopper questions are almost universally moronic.

[-TPM-]

Well I think most people that play PA speak atleast some english, but if that where an issue you could use mathmatical questions. Although that'd be easier to get around...

[Phil^]

i think you have misunderstood what a floodnet is. the previous posts cover it quite nicely.

a challenge-response thing would be pointless as once the bots join - the damage is done. challenging them all would result in the challenger being flooded off the network due to the number of queries sent.

cisco, ive not been able to do much work on it yet. i need an irc bot framework i can buid this upon first

[-TPM-]

Originally Posted by Phil^

i think you have misunderstood what a floodnet is. the previous posts cover it quite nicely.

a challenge-response thing would be pointless as once the bots join - the damage is done. challenging them all would result in the challenger being flooded off the network due to the number of queries sent.

cisco, ive not been able to do much work on it yet. i need an irc bot framework i can buid this upon first

It's true to say I've not witnessed it. But surely if you could quickly determine valid users as they joined, and kick those that aren’t that'd solve the problem. Or is it just the joining that causes the flood? (I imagined all of the joining bot's posting loads of messages)

[Phil^]

its the joining, spamming and the parting which is the problem. you cant quickly validate upwards of 30+ joins all at once by either messaging, whoising etc since you only flood yourself off in the process - or get caught in the anti-flood protection and have to wait for all the requests to go out of the queue.

[-TPM-]

If they got kicked for any message other than the yes/no though wouldn't that help?

What if you challenge them before the can enter? Make the room invite only.

[Phil^]

no since you have to then differentiate between random people who join, and the floodnet. plus not every flood net actually has the bots talk, or notice the channel etc. sometimes they just all join and part together a few times.
plus #planetarion is a public channel, making it invite only isnt a possibility - nor is leaving +r on.

[-TPM-]

Would a whois help if you didn't need to go through P? I know Zombie networks where mentioned. Maybe if this was paired with connection count within the last X second's it would al least cut it down.

[Phil^]

no, the whois would still be traffic that is required to be sent and recieved to and from the server for each and every join.

[-TPM-]

Yeah a whois would create traffic, but what I'm getting at is that a whois is'n't needed to get the IP info. It's actually included with the nick, your client just cleans it up.

[Phil^]

i know, the nick!ident@hostmask gets transmitted in the join message.

[pablissimo]

The challenge-response thing would work pretty well if coded straight into the ircd, would work like a constantly-changing key. You'd try and join the channel, you'd get a message (like the 'this channel requires a key' message but instead 'what is the capital of France?') and you use your answer as the key to join the chan.

Problem is that it

• Requires bot-stopping-but-human-usable questions which aren't trivial to write
• Prevents legitimate bots from joining the channel on part (unless the ircd keeps more state information and remembers that the user has authed at least once before)
• Requires actually arsing yourself to modify the ircd, and hoping your code alterations don't screw things

Doing it by invite as you say would potentially be a good idea, except that can you not invite even without having op privileges? In which case you could get one bot through the door who invites the next two who invites... which could be prevented by again adding more state to the bot that's meant to be controlling things, but that starts making it more complex and less elegant.

[-TPM-]

Originally Posted by Phil^

i know, the nick!ident@hostmask gets transmitted in the join message.

So that could be used to help right? Sure it wouldn't be the total answer, but every little helps

[-TPM-]

Originally Posted by pablissimo

The challenge-response thing would work pretty well if coded straight into the ircd, would work like a constantly-changing key. You'd try and join the channel, you'd get a message (like the 'this channel requires a key' message but instead 'what is the capital of France?') and you use your answer as the key to join the chan.

Problem is that it

• Requires bot-stopping-but-human-usable questions which aren't trivial to write
• Prevents legitimate bots from joining the channel on part (unless the ircd keeps more state information and remembers that the user has authed at least once before)
• Requires actually arsing yourself to modify the ircd, and hoping your code alterations don't screw things

Doing it by invite as you say would potentially be a good idea, except that can you not invite even without having op privileges? In which case you could get one bot through the door who invites the next two who invites... which could be prevented by again adding more state to the bot that's meant to be controlling things, but that starts making it more complex and less elegant.

I was thinking along these lines:
A user Joins and the bot kicks them and PM's them the the login question.
Or
The room is invite only and they pm the bot.
They answer the question, if correct the bot invites them in/dosen't kick them.

[Phil^]

again both are not feasible due to the additional traffic required to do them.
the best suggestion so far is along the lines that xtothez suggested, for the problem.
any bot would be limited in such a way that it has to be extremely careful not to flood itself off, therefore traffic is at a premium.
Furthermore the channel cannot be made invite only, nor can the bot kick people and ask them questions.

[-TPM-]

You can control your own traffic though. You could only initiate a new challenge every 2 seconds. That paired with the IP and maybe a mask check too would make the challenge even harder to get round.

[Phil^]

you dont understand
after 30 seconds or so the damage by the floodnet is already done. if 50 bots join at once, thats 100 seconds before every one of them is challenged.
then theres the problem when they part and rejoin again - 200 seconds. and so on.
the algorithm needs to be quick at identifying them, keep all traffic to a bare minimum ( meaning no challenge authentication ) , and lock the channel as fast as possible once its detected a floodnet in progress.
after the channels locked it can kick/ban at its leisure since no new ones can join.

[-TPM-]

No because those bot's arn't in the room for that time. You kick them, then challenge them. You could also keep a log of who's joined/left/kicked and if someone has too many ban them.

[Phil^]

yes, they are.
the bot is not a server, it cannot see people who join the network but not a channel.
it is essentially a regular client - all it has to go on are join , part, text and notice messages.

the idea of being able to message the bots with a challenge before they can join the room is limited to the server only. thats something they would have to do and its unlikely to happen imo.
you cannot message bots as they join the channel either, ive gone over the reasons why above.
as things are, a challenge based system just wont work im afraid.

[-TPM-]

Maybe you don't understand what I ment (or maybe I'm just totally missing something here ).

Yes the bot's join, but as soon as they do they're kicked. Then the challenge is sent via PM. So you don't have 50 bots sitting in the room. Maybe we should meet in IRC and talk about this, it might be easier....

[Phil^]

and the both the kick, and the challenge are extra traffic which is not necessary. PLUS people dont exactly appreciate being kicked, and further more a kick does not prevent then rejoining anyway in which case you have to either ban or rekick them again meaning even more traffic.
i understand what you meant, but it just does not fit the required task.

[-TPM-]

Hmm yeah I see what you mean. Invite only would work though wouldn't it? People just PM the bot for the invite and are challenged there. Invites would have to be limited to op's though as pablissimo said. The worst they could do then would be to crash the bot, which could just restart....

[xtothez]

I've been discussing this issue with Phil^ and we've come up with a basic framework for the service. This assumes the software is running as a services server (which would be the most logical way to approach this problem).

Each network client has a points total (starting at zero) modified by:

• Evaluating its nickname/ident/realname to a pattern upon connection. i.e: CCSVNNN where C = consonant, V = vowel, N = number, S = symbol (such as [ ] - _| ). These patterns are then compared to other clients that have connected in the previous X minutes. If there is a match, both clients have their points total incremented by an amount. This amount is based on how many matches are found, making points increase exponentially as more and more clients join with similar patterns.
• Joining an open channel* within X minutes of other clients with points above a certain threshold. Again, this increases exponentially.
• Joining a channel that appears to be a botnet control channel (they often have commands in topic that bots will run upon joining).
• All of a clients points are cleared upon authentication with P. They will also be immune to further detection.

Overall network 'alert level', incremented by:

• increase in total network clients in last X minutes
• large increase in clients on a specific server
• glining large amounts of users via proxyscan/excessive connections in last X minutes

Alert level drops gradually over time.

Automated reactions would depend upon the overall system alert level, and range from the service temporarily joining a 'targeted' channel to check for spam text and set protective modes (+mr), to automatic glines of suspected clients over a certain points threshold. Obviously we need to decide on less vague formulae for these points calculations that are both effective at detecting real bots but produce a minimal amount of false positives.

If anyone has additional comments on the above, or further ideas to add, post away...


* Channel has no keyed, invite-only or secret modes.

[Igloo]

You do know what the easiest thing would be?

Get Pea a white line (orwhatever netgamers calls them) basicaly a lagless connection to the server.

So it can detect, then kick and then ban the host.

So then Pea can have its delays taken out of it. And thus make its reactions instantainous, especially with the new "its all loaded into ram" versions we have.

So go speak to the opers, get a white line, Get pea re-coded, and see what happens

(i can give a guess of 99% that you will get very few that make it through the gate now).

[Phil^]

if you can get a floodless connection to the server - you might as well make it a server bot which is the line of thinking me and xtothez went down in our discussion.
as a server bot it would be able to see more then a normal channel bot could and perhaps act before they even joined a channel

[xtothez]

I have a couple of days off work later this week when I'll starting putting together some prototype code (initially for logging bot activity over the next few months). It will run as a network service, that's by far the most practical method for this kind of program.