Building a PA Dump Ripper

[The Chip]

I am not pleased with the Planetarion Decision to withhold full database dumps(the ones inlcuing ministers and galaxy pictures) to only the "official toolkit"

This thread is NOT about wiether that is right or wrong. Please don't become off topic.

The basic topic of ripping the data from Mit's site has been discussed. This is not a viable long term solution:
1) Easy to detect a single host doing thousands of queries a day.
2) overly abusive IPs can and will be banned.
3) Duplication of effort (many poeple do it, and his site will be hammered, a loss for everyone.)
4) many more beyond the scope of this discussion.

Therefore, the best solution in my opinion is a completely distributed system with a central collection point at which the public can freely download the complete dump.

Two Main Parts:

1) Create a Centeral Server. Could just be with PHP, or Perl, or any other system over HTTP. (or it could be somthing else, just depends on who codes it)

2) Create a Client application that many hundreds(or thousands) or people can run on their system.

I will start with the client.

The client must be able to run on as many systems as possible. This means at least a Linux(high-bandwidth) and Windows(popularity) client. It must communicate with both the PA Toolkit site, and with our central Server. The data it suuplies should not be trusted as an absolute . Each query should be valiated at least twice. If a difference is seen, it should be checked again by a third client(or more). Clients that are repeatidly cuaght feeding bad data should be automaticly removed or ignored by the server. The client will request a set of queries to make. The server will respond with "I need galaxies 5:5:, 24:3:, 11:1.. etc". The client will report each query as it makes them back to the Central Server. The client must randomly pace its queries over a resonable rate. A single client should not request objects 24 hours day. It should be spaced out to simulate a human browser. The client should run in the background(windows), with little or no intervention from the end user. The client should have the ability to remotely upgrade it parsing routines in the case the PA Toolkit changes its layout.

The Server has two main tasks.
First, keeping track of the clients, how many are online, what they are scanning, and which clients have been tagged as illegitimate(bad/corrupted data). The servers goal should be to provide a complete db dump say once per 24 hours. It should allocate when each galaxy is updated, and assign that to clients. It must keep track and make sure data is validated in a timely matter. It should also uniquely identify client. weither this is a username and password setup, or just a plain unique id, it doesn't matter. Perhaps a web of trust to authenticate new clients? The server should also provide updated parsing routines for the clients. The second main task of the server is combining the data into one final dump. Care should be taken to not allow any corrupted data(i.e. a modified client trying to feed bad data into the server).

opinions/suggestions??

anyone interested?

thanks for listening.

-chip

[W]

Originally posted by The Chip
I am not pleased with the Planetarion Decision to withhold full database dumps(the ones inlcuing ministers and galaxy pictures) to only the "official toolkit"

This thread is NOT about wiether that is right or wrong. Please don't become off topic.

The basic topic of ripping the data from Mit's site has been discussed. This is not a viable long term solution:
1) Easy to detect a single host doing thousands of queries a day.
2) overly abusive IPs can and will be banned.
3) Duplication of effort (many poeple do it, and his site will be hammered, a loss for everyone.)
4) many more beyond the scope of this discussion.

Therefore, the best solution in my opinion is a completely distributed system with a central collection point at which the public can freely download the complete dump.

Two Main Parts:

1) Create a Centeral Server. Could just be with PHP, or Perl, or any other system over HTTP. (or it could be somthing else, just depends on who codes it)

2) Create a Client application that many hundreds(or thousands) or people can run on their system.

I will start with the client.

The client must be able to run on as many systems as possible. This means at least a Linux(high-bandwidth) and Windows(popularity) client. It must communicate with both the PA Toolkit site, and with our central Server. The data it suuplies should not be trusted as an absolute . Each query should be valiated at least twice. If a difference is seen, it should be checked again by a third client(or more). Clients that are repeatidly cuaght feeding bad data should be automaticly removed or ignored by the server. The client will request a set of queries to make. The server will respond with "I need galaxies 5:5:, 24:3:, 11:1.. etc". The client will report each query as it makes them back to the Central Server. The client must randomly pace its queries over a resonable rate. A single client should not request objects 24 hours day. It should be spaced out to simulate a human browser. The client should run in the background(windows), with little or no intervention from the end user. The client should have the ability to remotely upgrade it parsing routines in the case the PA Toolkit changes its layout.

The Server has two main tasks.
First, keeping track of the clients, how many are online, what they are scanning, and which clients have been tagged as illegitimate(bad/corrupted data). The servers goal should be to provide a complete db dump say once per 24 hours. It should allocate when each galaxy is updated, and assign that to clients. It must keep track and make sure data is validated in a timely matter. It should also uniquely identify client. weither this is a username and password setup, or just a plain unique id, it doesn't matter. Perhaps a web of trust to authenticate new clients? The server should also provide updated parsing routines for the clients. The second main task of the server is combining the data into one final dump. Care should be taken to not allow any corrupted data(i.e. a modified client trying to feed bad data into the server).

opinions/suggestions??

anyone interested?

thanks for listening.

-chip

This is overly complex compared to the (simple) task at hand; to produce a database to interested clients. If you're really interested in peer to peer distributed nets with trust etc, there are actual research being done, trough open source projects, that you're free to join.


[Edit] I just realized what you're trying to do. Ie browsing gal pages or whatever on the users accounts. I believe both me and snow have had code to do this previously. Mine is for sure lost among the 5k useless projects a year I write, but I think snow put some sources on the web.

[The Chip]

Originally posted by W
This is overly complex compared to the (simple) task at hand; to produce a database to interested clients. If you're really interested in peer to peer distributed nets with trust etc, there are actual research being done, trough open source projects, that you're free to join.

It is overly complete because a simple approach would be blocked.(ie ripping 15k a day from one or two clients). Yes I admit some of the areas are overly complex, but the basic concept of a distributed ripper has merit I think.

[queball]

What's to stop me starting 15 accounts and giving you dodgy information about a galaxy? And then if I get caught, starting another 15? etc
Can you take advantage of proxy servers? Could the "client" just be a Squid server, restricted to incoming connections from the server, and outgoing connections to Mit's site, with some User-Agent foo? And the main server would have a list of accounts, each associated with one IP.

[Gayle29uk]

1. Gal banner spyware is one of the main reasons I always used pilkara. Include banners in a site and you can forget me and many others ever using it.

2. I don't care if my target is MoC/MoW/whatever, if I'm going to attack I'll do it anyway.

Looks like the usual fuss over nothing to me.

[The Chip]

Originally posted by Gayle29uk
1. Gal banner spyware is one of the main reasons I always used pilkara. Include banners in a site and you can forget me and many others ever using it.

2. I don't care if my target is MoC/MoW/whatever, if I'm going to attack I'll do it anyway.

Looks like the usual fuss over nothing to me.

this isn't a website to view stuff. this is a way to access that information if you choose to.

Please pay attention to what i said, this is not about weither this is good or bad or whatnot, this is about the technicalities. There are plently of other theads on these forums for your thoughts on the Planetarion decision on the matter.

[The Chip]

Originally posted by queball
What's to stop me starting 15 accounts and giving you dodgy information about a galaxy? And then if I get caught, starting another 15? etc
Can you take advantage of proxy servers? Could the "client" just be a Squid server, restricted to incoming connections from the server, and outgoing connections to Mit's site, with some User-Agent foo? And the main server would have a list of accounts, each associated with one IP.

That might be more viable.... but the problem is still getting enough proxies....

[Gayle29uk]

Originally posted by The Chip
this isn't a website to view stuff. this is a redundant way to access that information if you choose to.

I made a slight change.

Please pay attention to what i said, this is not about weither this is good or bad or whatnot, this is about the technicalities. There are plently of other theads on these forums for your thoughts on the Planetarion decision on the matter.

Please pay attention to what i said, if you want to write a distributed system of some sort then you'd be better doing something useful as opposed to something as utterly redundant as this. It serves no real purpose so why would anyone want to run the client? That's your first problem before you even THINK about design/coding.

[The Chip]

Originally posted by Gayle29uk
[b]I made a slight change.

Please pay attention to what i said, if you want to write a distributed system of some sort then you'd be better doing something useful as opposed to something as utterly redundant as this. It serves no real purpose so why would anyone want to run the client? That's your first problem before you even THINK about design/coding.

I believe there is a signifigant group of people, whom also have signifigant resources, that are interfested in using the information provided by these database dumps, even if you feel they are redundent. (I still don't see WHY having redundent, or simular sites is bad?)... but that is off topic, go post your stuff in the other threads, thanks.

[Gayle29uk]

Originally posted by The Chip
I believe there is a signifigant group of people, whom also have signifigant resources, that are interfested in using the information provided by these database dumps, even if you feel they are redundent. (I still don't see WHY having redundent, or simular sites is bad?)... but that is off topic, go post your stuff in the other threads, thanks.

On one hand you have a guaranteed (as much as anything in PA is) accurate and up to date site, on the other hand you have a site where data may be incomplete or out of date. Again I ask, what does anyone gain from this? Data on ministers and gal banners seem a poor trade off for dodgy data to me.

If you REALLY wanted to do a site with all the details then the logical way would be to have a text browser (save his bandwidth) go through MITs site once a day for minister positions and gal banners. I'd estimate this will be 400-600 pages max per day and you have all the info you need because everything else is already in the dump!

[edit]*cough* maths *cough* [/edit]

[queball]

Gayle, http://pirate.planetarion.com/showth...hreadid=159315

[pablissimo]

Originally posted by The Chip
I believe there is a signifigant group of people, whom also have signifigant resources, that are interfested in using the information provided by these database dumps, even if you feel they are redundent. (I still don't see WHY having redundent, or simular sites is bad?)... but that is off topic, go post your stuff in the other threads, thanks.

The point is that such concerns are part of software engineering, and it's fairly obvious you're choosing a way of doing this that is neither efficient nor has signifigant gains for the resources you're putting into it. If you do a full manual dump along the lines you're suggesting, the data you're harvesting will be out of date, and as such unreliable when it comes to being used as a tool for your PA gamer.

I like Gayle's method more to be honest.

[Gayle29uk]

Originally posted by queball
Gayle, http://pirate.planetarion.com/showth...hreadid=159315

Been there, done that too

[queball]

Originally posted by Gayle29uk

If you REALLY wanted to do a site with all the details then the logical way would be to have a text browser (save his bandwidth) go through MITs site once a day for minister positions and gal banners. I'd estimate this will be 40-60 pages max per day and you have all the info you need because everything else is already in the dump!

This is what chip is suggesting, except automated software instead of a textmode browser.

[Gayle29uk]

Originally posted by queball
This is what chip is suggesting, except automated software instead of a textmode browser.

He's suggesting something unnecessarily complex though, all this business about people running clients is pointless when for a daily dump of MITs site you're looking at 5-600 pages max (500 pages being a 5000 player universe). A simple PHP parser (no images remember) could do it easily, you could do one every 10 seconds to make sure you're playing nice and it'd hardly touch his bandwidth.

[queball]

Originally posted by Gayle29uk
He's suggesting something unnecessarily complex though, all this business about people running clients is pointless when for a daily dump of MITs site you're looking at 5-600 pages max (500 pages being a 5000 player universe). A simple PHP parser (no images remember) could do it easily, you could do one every 10 seconds to make sure you're playing nice and it'd hardly touch his bandwidth.

Starting with the assumption that such a tool would be IP banned, this project is the best alternative.

[Gayle29uk]

Originally posted by queball
Starting with the assumption that such a tool would be IP banned, this project is the best alternative.

Running the data gathering part on a computer with a dynamic IP then uploading it to the server would get round that. They're not going to block entire IP ranges after all.

[queball]

Because PA have never blocked IP ranges?
How about if they put a limit per IP of 200 pages per day?

[Gayle29uk]

Originally posted by queball
Because PA have never blocked IP ranges?
How about if they put a limit per IP of 200 pages per day?

Run 199, disconnect, rinse and repeat

[queball]

The reason the context of the tool is important is because it defines the specification. A decent alternative toolkit site should be reliable, up to date, and difficult to stop.

There are various problems with relying on a dialup user. There could be thousands of pages depending on how many galaxies there are. At one every ten seconds that's two hours or more, and it would be make an obvious pattern. I'd say any information more than a day old is unusable. Ideally each client would access fifty or so pages, in one hour per day. We then need twenty clients, and there are at least this number on http://www.antiproxy.org/.

Edit: replace hour with day and fix numbers

[Gayle29uk]

Originally posted by queball
The reason the context of the tool is important is because it defines the specification. A decent alternative toolkit site should be reliable, up to date, and difficult to stop.

There are various problems with relying on a dialup user. There could be thousands of pages depending on how many galaxies there are. At one every ten seconds that's two hours or more, and it would be make an obvious pattern. I'd say any information more than an hour old is unusable. Ideally each client would access thirty or so pages over five minutes, maybe twice an hour. We then need two hundred clients, and there are at least this number on http://www.antiproxy.org/.

But the only information needed from MITs site is the minister/gal banner data which can be updated once a day and still be regarded as current. The rest of the info is all legitimately avaiolable through the dump so why not get it from there?

[queball]

edited

[Gayle29uk]

Ooh, forgot something else...

There's no way (R9 at least) that theres going to be thousands of pages. 10k users for the next round is just NOT gonna happen!

Estimate 4-6k players and that gives 4-600 pages per day that need parsing

[Epcylon]

So you propose that we design a system that is non-scalable and requires a complete rewrite for R10...?

[Gayle29uk]

Originally posted by Epcylon
So you propose that we design a system that is non-scalable and requires a complete rewrite for R10...?

So you propose instead of a system that can be written, tested, and implemented within a day you write a massively over specced system for R9 that could well be utterly useless for R10 because of all the changes Spinner is making?

Writing anything with R10 in mind is pointless, there's a good chance you'll be completely wasting your time if it's changed as extensively as proposed.

[MT]

I dont get the problem.. how many people still do isolated one off attacks? Galaxy attacks, you just load the gal page once, see who is minister or whatever, no worries.

Planning for R10 is pointless, PA will be gone by then, and we will all be playing www.hattrick.org

[BesigedB]

is hattrick FSAS's new venture?

[Mit]

and what would u lot do if i chose not to display minister info or gal banners at all...

[Pineapple]

Originally posted by Mit
and what would u lot do if i chose not to display minister info or gal banners at all...

well your tool would then be no different than what was on offer... so people would have two options...

a) keep using the standard dumps
b) ask the crew to make a public dump with the extra info

[Gayle29uk]

Originally posted by Mit
and what would u lot do if i chose not to display minister info or gal banners at all...

To be honest it'd make me more likely to use it.

I'd prefer to use a non-allied source of information but not one with gal banners, they slow things down and potentially provide too much information to the gal I'm looking at.

To be completely honest, the best use of the info (imo) is to create a 'banner gallery' that people can look through if they're interested in the baners. Something original that some people would find interesting but doesn't give any information away

[inf]

who actually cares about the extra information? it's not really that useful.

Originally posted by Gayle29uk
1. Gal banner spyware is one of the main reasons I always used pilkara. Include banners in a site and you can forget me and many others ever using it.

2. I don't care if my target is MoC/MoW/whatever, if I'm going to attack I'll do it anyway.

Looks like the usual fuss over nothing to me.

Why do you even care about "spyware"?
Noone (and I do mean noone) cares about some randome noobies location,so why should they care about some randome girl who started playing a short while ago?

[Gayle29uk]

Originally posted by DarthDaddy
Why do you even care about "spyware"?
Noone (and I do mean noone) cares about some randome noobies location,so why should they care about some randome girl who started playing a short while ago?

Are you really that dumb or is it an act? Go crawl back under your rock.

[pablissimo]

Originally posted by inf
who actually cares about the extra information? it's not really that useful.

Amen.

[mist]

if the amount of open proxies i've seen on irc is anything to go by there's no need to write a client, just to get a list of them off the web. i think your plan is much more complicated than it needs to be

-mist

if this is done on a server on ntl or some such it will use their web proxy so no way will they ban that ip

[meglamaniac]

You're all walking a very thin line between what is allowable here, and quite frankly what you're getting worked up about is trivial.
Let's assume I was actually going to play round 9 for a start. If I was to use any form of search, then I wouldn't want galaxy banners anyway (waste of time and bandwidth) and the minister info is trivial. I could get it in seconds through the game.

I wouldn't be surprised if a rule was inserted into the EULA (or whatever you want to call it) for Round 9 that would make bot access of official planetarion resources in addition to the game itself a deleteable offense, and now you've drawn addequate attention to your plans I'd expect a few of the crew to be looking over this thread in the next few days to make thier decision.

Oh and chip, don't be so naive as to try and delete it or anything - it's allready been copied.

[W]

Originally posted by meglamaniac
You're all walking a very thin line between what is allowable here, and quite frankly what you're getting worked up about is trivial.
Let's assume I was actually going to play round 9 for a start. If I was to use any form of search, then I wouldn't want galaxy banners anyway (waste of time and bandwidth) and the minister info is trivial. I could get it in seconds through the game.

I wouldn't be surprised if a rule was inserted into the EULA (or whatever you want to call it) for Round 9 that would make bot access of official planetarion resources in addition to the game itself a deleteable offense, and now you've drawn addequate attention to your plans I'd expect a few of the crew to be looking over this thread in the next few days to make thier decision.

Oh and chip, don't be so naive as to try and delete it or anything - it's allready been copied.

"Hahahahaha, I had a tape recorder! I'm gonna rat on all of you now! You're all going to jain muahahaha!"

Err...

Way back in round TWO, I pestered the creators to include this info in the dump. By the end of the round, I had pretty fool proof systems for not only bot'ing safely, but multiing safely too. If someone wants this info bad enough, they'll get it, alone if they're forced to by the creators. If it wasn't such a big deal, it would already be included. this is a matter of principle, and always have been.

[meglamaniac]

Organising what ammounts to a Distributed DoS attack on a server run for the game by a volunteer (who isn't even connected to the game in any way, afaik) isn't the answer.
You may assume because it's an official part of the game it's got some nice high bandwidth lines and is hosted by jolt. If you do, you're wrong on both counts.

This wonderful masterplan would simply result in the server being taken offline and there being no route at all for you to get this precious data you all so desperately seem to want.

Consider the consequences of your actions before bowling headlong down the slope with the whole 'badly done for consumer' ticket.

[inf]

You could take down the forums with a dialup modem.

MIT has sex with goats on a regular basis.

[MT]

No offence, but PA have absolutely no way of tracking/catching a decent effort to get this data. The estimated numbers for rd 9 are so fking minusculely small, to munge the data from any web source will be piddle. Remember that 'PA Dumps' didnt exist from the start, yet we still had PBOT, and that was getting all data over PA's 2 Mbps link.

Its not like they have any half intelligent admins left anyways that could even start to catch this.

<Kloopy> And besides, I'm on the coding team this time, it'll be coded efficiently so it wont need a complete server farm to run on.

Humour++;

Whats the big fking deal of putting in GC/Minister info into the db dump. Its completely wanking illogical. I cant wait to see how they shoot themselves in the foot next tbh,

[Mit]

http://pirate.planetarion.com/showth...hreadid=159635 <-- READ

[W]

Originally posted by meglamaniac
Organising what ammounts to a Distributed DoS attack on a server run for the game by a volunteer (who isn't even connected to the game in any way, afaik) isn't the answer.
You may assume because it's an official part of the game it's got some nice high bandwidth lines and is hosted by jolt. If you do, you're wrong on both counts.

This wonderful masterplan would simply result in the server being taken offline and there being no route at all for you to get this precious data you all so desperately seem to want.

Consider the consequences of your actions before bowling headlong down the slope with the whole 'badly done for consumer' ticket.

I assume he meanth to get it from the gal pages directly. Might as well when he's running a client from each player involved.

[MT]

Originally posted by Mit
http://pirate.planetarion.com/showth...hreadid=159635 <-- READ

What relevance does that have to this thread? I'm tempted to delete it, as this thread is about a distributed or otherwise way of getting the extra information. Its not about the decision. Posts must be relevant to the thread in future.

[Mit]

Originally posted by MT
What relevance does that have to this thread? I'm tempted to delete it, as this thread is about a distributed or otherwise way of getting the extra information. Its not about the decision. Posts must be relevant to the thread in future.

"distributed or otherwise way...." because that is the 'otherwise' way of getting the data it saves everyone effort and IS relevant because its about that data they want.

[The Chip]

Originally posted by meglamaniac
Oh and chip, don't be so naive as to try and delete it or anything - it's allready been copied.

lol.

im scared now.

they might *shudder* ban me for talking *about* an *idea* to get *public* information.

okay meglamaniac, go ahead, tell the whole world that chip was the one who started this thread.

Kittens with Mittens!

[Kumnaa]

Originally posted by Idi
Kittens with Mittens!

we dont want your kind here, go back to the SA forums!

[Ramihyn]

Any alliance which really wants this data, will get it cheaply and undetectable. You just ask a few people to connect through a transparent proxy which filters the galaxypage url. A transparent proxy is easy to do - parsing of html pages to collect this kind of info, was already done long ago before dumps existed. The data will be as up to date as the last time a member looked at a gal and it doesnt put any additional bandwidth on any server. You can always create a list of "outdated" galaxy coordinates on some page, so the people can click on a coord to automagically update the info (no you cant detect that with some silly http header parsing like looking for referers). Just think about the multiple other uses you can do with such a transparent filtering/parsing proxy ... frightening hm ? :/

btw. there are some uses of the additional data - if you dont know how to use something, it doesnt necessarily mean that there is no use for it.

ofc all this is pure theory and has nothing to do with reality

ps : isn't it annoying how much time and effort is wasted into this topic instead of enhancing the game for everyone? (talking about all sides here)